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Preface 



There are many textbooks that offer an introduction to artificial intelligence (AI). This text has 
five principal features that together distinguish it from other texts. 

1 . Unified presentation of the field. 

Some texts are organized from a historical perspective, describing each of the major 
problems and solutions that have been uncovered in 40 years of AI research. Although 
there is value to this perspective, the result is to give the impression of a dozen or so barely 
related subfields, each with its own techniques and problems. We have chosen to present 
AI as a unified field, working on a common problem in various guises. This has entailed 
some reinterpretation of past research, showing how it fits within a common framework 
and how it relates to other work that was historically separate. It has also led us to include 
material not normally covered in AI texts. 

2. Intelligent agent design. 

The unifying theme of the book is the concept of an intelligent agent. In this view, the 
problem of AI is to describe and build agents that receive percepts from the environment 
and perform actions. Each such agent is implemented by a function that maps percepts 
to actions, and we cover different ways to represent these functions, such as production 
systems, reactive agents, logical planners, neural networks, and decision-theoretic systems. 
We explain the role of learning as extending the reach of the designer into unknown environ- 
ments, and show how it constrains agent design, favoring explicit knowledge representation 
and reasoning. We treat robotics and vision not as independently defined problems, but 
as occurring in the service of goal achievement. We stress the importance of the task 
environment characteristics in determining the appropriate agent design. 

3. Comprehensive and up-to-date coverage. 

We cover areas that are sometimes underemphasized, including reasoning under uncer- 
tainty, learning, neural networks, natural language, vision, robotics, and philosophical 
foundations. We cover many of the more recent ideas in the field, including simulated 
annealing, memory-bounded search, global ontologies, dynamic and adaptive probabilistic 
(Bayesian) networks, computational learning theory, and reinforcement learning. We also 
provide extensive notes and references on the historical sources and current literature for 
the main ideas in each chapter. 

4. Equal emphasis on theory and practice. 

Theory and practice are given equal emphasis. All material is grounded in first principles 
with rigorous theoretical analysis where appropriate, but the point of the theory is to get the 
concepts across and explain how they are used in actual, fielded systems. The reader ofthis 
book will come away with an appreciation for the basic concepts and mathematical methods 
of AI, and also with an idea of what can and cannot be done with today's technology, at 
what cost, and using what techniques. 

5. Understanding through implementation. 

The principles of intelligent agent design are clarified by using them to actually build agents. 
Chapter 2 provides an overview of agent design, including a basic agent and environment 
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project. Subsequent chapters include programming exercises that ask the student to add >. 
capabilities to the agent, making it behave more and more interestingly and (we hope) 
intelligently. Algorithms are presented at three levels of detail: prose descriptions and ! 
pseudo-code in the text, and complete Common Lisp programs available on the Internet or 
on floppy disk. All the agent programs are interoperable and work in a uniform framework 
for simulated environments. 

This book is primarily intended for use in an undergraduate course or course sequence. It 
can also be used in a graduate-level course (perhaps with the addition of some of the primary 
sources suggested in the bibliographical notes). Because of its comprehensive coverage and the 
large number of detailed algorithms, it is useful as a primary reference volume for AI graduate 
students and professionals wishing to branch out beyond their own subfield. We also hope that 
AI researchers could benefit from thinking about the unifying approach we advocate. 

The only prerequisite is familiarity with basic concepts of computer science (algorithms, 
data structures, complexity) at a sophomore level. Freshman calculus is useful for understanding 
neural networks and adaptive probabilistic networks in detail. Some experience with nonnumeric 
programming is desirable, but can be picked up in a few weeks study. We provide implementations 
of all algorithms in Common Lisp (see Appendix B), but other languages such as Scheme, Prolog, 
Smalltalk, C++, or ML could be used instead. 

Overview of the book 

The book is divided into eight parts. Part I, "Artificial Intelligence," sets the stage for all the others, 
and offers a view of the AI enterprise based around the idea of intelligent agents — systems that 
can decide what to do and do it. Part II, "Problem Solving," concentrates on methods for deciding 
what to do when one needs to think ahead several steps, for example in navigating across country 
or playing chess. Part III, "Knowledge and Reasoning," discusses ways to represent knowledge 
about the world — how it works, what it is currently like, what one's actions might do — and how 
to reason logically with that knowledge. Part IV, "Acting Logically," then discusses how to 
use these reasoning methods to decide what to do, particularly by constructing plans. Part V, 
"Uncertain Knowledge and Reasoning," is analogous to Parts HI and IV, but it concentrates on 
reasoning and decision-making in the presence of uncertainty about the world, as might be faced, 
for example, by a system for medical diagnosis and treatment. 

Together, Parts II to V describe that part of the intelligent agent responsible for reaching 
decisions. Part VI, "Learning," describes methods for generating the knowledge required by these 
decision-making components; it also introduces a new kind of component, the neural network, 
and its associated learning procedures. Part VII, "Communicating, Perceiving, and Acting," 
describes ways in which an intelligent agent can perceive its environment so as to know what is 
going on, whether by vision, touch, hearing, or understanding language; and ways in which it can 
turn its plans into real actions, either as robot motion or as natural language utterances. Finally, 
Part VIII, "Conclusions," analyses the past and future of AI, and provides some light amusement 
by discussing what AI really is and why it has already succeeded to some degree, and airing the 
views of those philosophers who believe that AI can never succeed at all. 
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Using this book 

This is a big book; covering all the chapters and the projects would take two semesters. You will 
notice that the book is divided into 27 chapters, which makes it easy to select the appropriate 
material for any chosen course of study. Each chapter can be covered in approximately one week. 
Some reasonable choices for a variety of quarter and semester courses are as follows: 

• One-quarter general introductory course: 

Chapters 1, 2, 3, 6, 7, 9, 1 1, 14, 15, 18, 22. 

• One-semester general introductory course: 

Chapters 1, 2, 3, 4, 6, 7, 9, 11, 13, 14, 15, 18, 19, 22, 24, 26, 27. 

• One-quarter course with concentration on search and planning: 

Chapters 1, 2, 3, 4, 5, 6, 7, 9, 1 1, 12, 13. 

• One-quarter course with concentration on reasoning and expert systems: 

Chapters 1,2, 3, 6, 7, 8,9, 10, 11, 14, 15, 16. 

• One-quarter course with concentration on natural language: 

Chapters 1, 2, 3, 6, 7, 8, 9, 14, 15, 22, 23, 26, 27. 

• One-semester course with concentration on learning and neural networks: 

Chapters 1, 2, 3, 4, 6, 7, 9, 14, 15, 16, 17, 18, 19, 20, 21. 

• One-semester course with concentration on vision and robotics: 

Chapters 1, 2, 3, 4, 6, 7, 11, 13, 14, 15, 16, 17, 24, 25, 20. 

These sequences could be used for both undergraduate and graduate courses. The relevant parts 
of the book could also be used to provide the first phase of graduate specialty courses. For 
example, Part VI could be used in conjunction with readings from the literature in a course on 
machine learning. 

We have decided not to designate certain sections as "optional" or certain exercises as 
"difficult," as individual tastes and backgrounds vary widely. Exercises requiring significant 
programming are marked with a keyboard icon, and those requiring some investigation of the 
literature are marked with a book icon. Altogether, over 300 exercises are included. Some of 
them are large enough to be considered term projects. Many of the exercises can best be solved 
by taking advantage of the code repository, which is described in Appendix B. Throughout the 
book, important points are marked with a pointing icon. 

If you have any comments on the book, we'd like to hear from you. Appendix B includes 
information on how to contact us. 
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Part I 



ARTIFICIAL INTELLIGENCE 



The two chapters in this part introduce the subject of Artificial Intelligence or AI 
and our approach to the subject: that AI is the study of agents that exist in an 
environment and perceive and act. 




Section 1 .2. The Foundations of Artificial Intelligence 



and subtracting machine called the Pascaline. Leibniz improved on this in 1694, building a 
mechanical device that multiplied by doing repeated addition. Progress stalled for over a century 
until Charles Babbage ( 1792-187 1 ) dreamed that logarithm tables could be computed by machine. 
He designed a machine for this task, but never completed the project. Instead, he turned to the 
design of the Analytical Engine, for which Babbage invented the ideas of addressable memory, 
stored programs, and conditional jumps. Although the idea of programmable machines was 
not new — in 1805, Joseph Marie Jacquard invented a loom that could be programmed using 
punched cards — Babbage’s machine was the first artifact possessing the characteristics necessary 
for universal computation. Babbage's colleague Ada Lovelace, daughter of the poet Lord Byron, 
wrote programs for the Analytical Engine and even speculated that the machine could play chess 
or compose music. Lovelace was the world's first programmer, and the first of many to endure 
massive cost overruns and to have an ambitious project ultimately abandoned. 1 1 Babbage's basic 
design was proven viable by Doron Swade and his colleagues, who built a working model using 
only the mechanical techniques available at Babbage's time (Swade, 1993). Babbage had the 
right idea, but lacked the organizational skills to get his machine built. 

AI also owes a debt to the software side of computer science, which has supplied the 
operating systems, programming languages, and tools needed to write modern programs (and 
papers about them). But this is one area where the debt has been repaid: work in AI has pioneered 
many ideas that have made their way back to "mainstream" computer science, including time 
sharing, interactive interpreters, the linked list data type, automatic storage management, and 
some of the key concepts of object-oriented programming and integrated program development 
environments with graphical user interfaces. 

Linguistics (1957-present) 

In 1957, B. F. Skinner published Verbal Behavior. This was a comprehensive, detailed account 
of the behaviorist approach to language learning, written by the foremost expert in the field. But 
curiously, a review of the book became as well-known as the book itself, and served to almost kill 
off interest in behaviorism. The author of the review was Noam Chomsky, who hadjust published 
a book on his own theory, Syntactic Structures. Chomsky showed how the behaviorist theory did 
not address the notion of creativity in language — it did not explain how a child could understand 
and make up sentences that he or she had never heard before. Chomsky's theory — based on 
syntactic models going back to the Indian linguist Panini (c. 350 B.C.) — could explain this, and 
unlike previous theories, it was formal enough that it could in principle be programmed. 

Later developments in linguistics showed the problem to be considerably more complex 
than it seemed in 1957. Language is ambiguous and leaves much unsaid. This means that 
understanding language requires an understanding of the subject matter and context, not just an 
understanding of the structure of sentences. This may seem obvious, but it was not appreciated 
until the early 1 960s. Much of the early work in knowledge representation (the study of how to 
put knowledge into a form that a computer can reason with) was tied to language and informed 
by research in linguistics, which was connected in turn to decades of work on the philosophical 
analysis of language. 

1 1 She also gave her name to Ada, the U.S. Department of Defense's all-purpose programming language. 



INTRODUCTION 



In which we try to explain why we consider artificial intelligence to be a subject most 
worthy of study, and in which we try to decide what exactly it is, this being a good 
thing to decide before embarking. 



Humankind has given itself the scientific name homo sapiens — man the wise — because our 
mental capacities are so important to our everyday lives and our sense of self. The field of 
artificial intelligence, or AI, attempts to understand intelligent entities. Thus, one reason to 
study it is to learn more about ourselves. But unlike philosophy and psychology, which are 
also concerned with intelligence, AI strives to build intelligent entities as well as understand 
them. Another reason to study AI is that these constructed intelligent entities are interesting and 
useful in their own right. AI has produced many significant and impressive products even at this 
early stage in its development. Although no one can predict the future in detail, it is clear that 
computers with human-level intelligence (or better) would have a huge impact on our everyday 
lives and on the future course of civilization. 

AI addresses one of the ultimate puzzles. How is it possible for a slow, tiny brain, whether 
biological or electronic, to perceive, understand, predict, and manipulate a world far larger and 
more complicated than itself? How do we go about making something with those properties? 
These are hard questions, but unlike the search for faster-than-lighttravel or an antigravity device, 
the researcher in AI has solid evidence that the quest is possible. All the researcher has to do is 
look in the mirror to see an example of an intelligent system. 

AI is one of the newest disciplines. It was formally initiated in 1956, when the name 
was coined, although at that point work had been under way for about five years. Along with 
modem genetics, it is regularly cited as the "field I would most like to be in" by scientists in other 
disciplines. A student in physics might reasonably feel that all the good ideas have already been 
taken by Galileo, Newton, Einstein, and the rest, and that it takes many years of study before one 
can contribute new ideas. AI, on the other hand, still has openings for a full-time Einstein. 

The study of intelligence is also one of the oldest disciplines. For over 2000 years, philoso- 
phers have tried to understand how seeing, learning, remembering, and reasoning could, or should. 
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be done. 1 The advent of usable computers in the early 1950s turned the learned but armchair 
speculation concerning these mental faculties into a real experimental and theoretical discipline. 
Many felt that the new "Electronic Super-Brains" had unlimited potential for intelligence. "Faster 
Than Einstein" was a typical headline. But as well as providing a vehicle for creating artificially 
intelligent entities, the computer provides a tool for testing theories of intelligence, and many 
theories failed to withstand the test — a case of "out of the armchair, into the fire." AI has turned 
out to be more difficult than many at first imagined, and modem ideas are much richer, more 
subtle, and more interesting as a result. 

AI currently encompasses a huge variety of subfields, from general-purpose areas such as 
perception and logical reasoning, to specific tasks such as playing chess, proving mathematical 
theorems, writing poetry, and diagnosing diseases. Often, scientists in other fields move gradually 
into artificial intelligence, where they find the tools and vocabulary to systematize and automate 
the intellectual tasks on which they have been working all their lives. Similarly, workers in AI 
can choose to apply their methods to any area of human intellectual endeavor. In this sense, it is 
truly a universal field. 



1.1 What is AI? 



We have now explained why AI is exciting, but we have not said what it is. We could just say, 
"Well, it has to do with smart programs, so let's get on and write some." But the history of science 
shows that it is helpful to aim at the right goals. Early alchemists, looking for a potion for eternal 
life and a method to turn lead into gold, were probably off on the wrong foot. Only when the aim 
changed, to that of finding explicit theories that gave accurate predictions of the terrestrial world, 
in the same way that early astronomy predicted the apparent motions of the stars and planets, 
could the scientific method emerge and productive science take place. 

Definitions of artificial intelligence according to eight recent textbooks are shown in Fig- 
ure 1.1. These definitions vary along two main dimensions. The ones on top are concerned 
with thought processes and reasoning, whereas the ones on the bottom address behavior. Also, 
the definitions on the left measure success in terms of human performance, whereas the ones : 
rationality on the right measure against an ideal concept of intelligence, which we will call rationality. A 

system is rational if it does the right thing. This gives us fourpossible goals to pursue in artificial 
intelligence, as seen in the caption of Figure 1.1. 

Historically, all four approaches have been followed. As one might expect, a tension exists 
between approaches centered around humans and approaches centered around rationality. 2 A 
human-centered approach must be an empirical science, involving hypothesis and experimental 



1 A more recent branch of philosophy is concerned with proving that AI is impossible. We will return to this interesting 
viewpoint in Chapter 26. 

2 We should point out that by distinguishing between human and rational behavior, we are not suggesting that humans 
are necessarily "irrational" in the sense of "emotionally unstable" or "insane." One merely need note that we often make 
mistakes; we are not all ch< gi ndm ter even though we may know all the rules of chess; and unfortunately, not 
everyone gets an A on the exam. Some systematic errors in human reasoning are cataloged by Kahneman et al. ( 1 982). 
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"The exciting new effort to make computers 
think . . . machines with minds, in the full 
and literal sense" (Haugeland, 1985) 

"[The automation of] activities that we asso- 
ciate with human thinking, activities such as 
decision-making, problem solving, learning 
. . .” (Bellman, 1978) 


"The study of mental faculties through the 
use of computational models" 

(Charniak and McDermott, 1985) 

"The study of the computations that make 
it possible to perceive, reason, and act" 
(Winston, 1992) 


"The art of creating machines that perform 
functions that require intelligence when per- 
formed by people" (Kurzweil, 1 990) 

"The study of how to make computers do 
things at which, at the moment, people are 
better" (Rich and Knight, 1991) 


"A field of study that seeks to explain and 
emulate intelligent behavior in terms of 
computational processes" (Schalkoff, 1 990) 
"The branch of computer science that is con- 
cerned with the automation of intelligent 
behavior" (Luger and Stubblefield, 1993) 



Figure 1.1 Some definitions of AI. They are organized into four categories: 



Systems that think like humans. 


Systems that think rationally. 


Systems that act like humans. 


Systems that act rationally. 



confirmation. A rationalist approach involves a combination of mathematics and engineering. 
People in each group sometimes cast aspersions on work done in the other groups, but the truth 
is that each direction has yielded valuable insights. Let us look at each in more detail. 



Acting humanly: The Turing Test approach 



TURING TEST 



NATURAL LANGUAGE 

processing 

KNOWLEDGE 

REPRESENTATION 

automated 

REASONING 

MACHINE LEARNING 



The Turing Test, proposed by Alan Turing (1950), was designed to provide a satisfactory 
operational definition of intelligence. Turing defined intelligent behavior as the ability to achieve 
human-level performance in all cognitive tasks, sufficient to fool an interrogator. Roughly 
speaking, the test he proposed is that the computer should be interrogated by a human via a 
teletype, and passes the test if the interrogator cannot tell if there is a computer or a human at the 
other end. Chapter 26 discusses the details of the test, and whether or not a computer is really 
intelligent if it passes. For now, programming a computer to pass the test provides plenty to work 
on. The computer would need to possess the following capabilities: 

0 natural language processing to enable it to communicate successfully in English (or some 
other human language); 

0 knowledge representation to store information provided before or during the interrogation; 

0 automated reasoning to use the stored information to answer questions and to draw new 
conclusions; 

0 machine learning to adapt to new circumstances and to detect and extrapolate patterns. 



Turing's test deliberately avoided direct physical interaction between the interrogator and the 
computer, because physical simulation of a person is unnecessary for intelligence. However, 
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TOTAL TDRING TEST the so-called total Turing Test includes a video signal so that the interrogator can test the 
subject's perceptual abilities, as well as the opportunity for the interrogator to pass physical 
objects "through the hatch." To pass the total Turing Test, the computer will need 

COMPUTER VISION <> computer vision to perceive objects, and 

ROBOTICS 0 robotics to move them about. 

Within AI, there has not been a big effort to try to pass the Turing test. The issue of acting 
like a human comes up primarily when AI programs have to interact with people, as when an 
expert system explains how it came to its diagnosis, or a natural language processing system has 
a dialogue with a user. These programs must behave according to certain normal conventions of 
human interaction in order to make themselves understood. The underlying representation and 
reasoning in such a system may or may not be based on a human model. 



Thinking humanly: The cognitive modelling approach 

If we are going to say that a given program thinks like a human, we must have some way of 
determining how humans think. We need to get inside the actual workings of human minds. 
There are two ways to do this: through introspection — trying to catch our own thoughts as they 
go by — or through psychological experiments. Once we have a sufficiently precise theory of 
the mind, it becomes possible to express the theory as a computer program. If the program's 
input/output and timing behavior matches human behavior, that is evidence that some of the 
program's mechanisms may also be operating in humans. For example, Newell and Simon, who 
developed GPS, the "General Problem Solver" (Newell and Simon, 1961), were not content to 
have their program correctly solve problems. They were more concerned with comparing the 
trace of its reasoning steps to traces of human subjects solving the same problems. This is in 
contrast to other researchers of the same time (such as Wang (I960)), who were concerned with 
getting the right answers regardless of how humans might do it. The interdisciplinary field of 
cognitive science cognitive science brings together computer models from AI and experimental techniques from 
psychology to try to construct precise and testable theories of the workings of the human mind. 

Although cognitive science is a fascinating field in itself, we are not going to be discussing 
it all that much in this book. We will occasionally comment on similarities or differences between 
AI techniques and human cognition. Real cognitive science, however, is necessarily based on 
experimental investigation of actual humans or animals, and we assume that the reader only has 
access to a computer for experimentation. We will simply note that AI and cognitive science 
continue to fertilize each other, especially in the areas of vision, natural language, and learning. 
The history of psychological theories of cognition is briefly covered on page 12. 



Thinking rationally: The laws of thought approach 

The Greek philosopher Aristotle was one of the first to attempt to codify "right thinking," that is, 
irrefutable reasoning processes. His famous syllogisms provided patterns for argument structures 
that always gave correct conclusions given correct premises. For example, "Socrates is a man; 
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all men are mortal; therefore Socrates is mortal." These laws of thought were supposed to govern 
the operation of the mind, and initiated the field of logic. 

The development of formal logic in the late nineteenth and early twentieth centuries, which 
we describe in more detail in Chapter 6, provided a precise notation for statements about all kinds 
of things in the world and the relations between them. (Contrast this with ordinary arithmetic 
notation, which provides mainly for equality and inequality statements about numbers.) By 1965, 
programs existed that could, given enough time and memory, take a description of a problem 
in logical notation and find the solution to the problem, if one exists. (If there is no solution, 
the program might never stop looking for it.) The so-called logicist tradition within artificial 
intelligence hopes to build on such programs to create intelligent systems. 

There are two main obstacles to this approach. First, it is not easy to take informal 
knowledge and state it in the formal terms required by logical notation, particularly when the 
knowledge is less than 100% certain. Second, there is a big difference between being able to 
solve a problem "in principle” and doing so in practice. Even problems with just a few dozen 
facts can exhaust the computational resources of any computer unless it has some guidance as to 
which reasoning steps to try first. Although both of these obstacles apply to any attempt to build 
computational reasoning systems, they appeared first in the logicist tradition because the power 
of the representation and reasoning systems are well-defined and fairly well understood. 



Acting rationally: The rational agent approach 

agent Acting rationally means acting so as to achieve one's goals, given one's beliefs. An agent isjust 

something that perceives and acts. (This may be an unusual use of the word, but you will get 
used to it.) In this approach, AI is viewed as the study and construction of rational agents. 

In the "laws of thought" approach to AI, the whole emphasis was on correct inferences. 
Making correct inferences is sometimes part of being a rational agent, because one way to act 
rationally is to reason logically to the conclusion that a given action will achieve one's goals, 
and then to act on that conclusion. On the other hand, correct inference is not all of rationality, 
because there are often situations where there is no provably correct thing to do, yet something 
must still be done. There are also ways of acting rationally that cannot be reasonably said to 
involve inference. For example, pulling one's hand off of a hot stove is a reflex action that is 
more successful than a slower action taken after careful deliberation. 

All the "cognitive skills" needed for the Turing Test are there to allow rational actions. Thus, 
we need the ability to represent knowledge and reason with it because this enables us to reach 
good decisions in a wide variety of situations. We need to be able to generate comprehensible 
sentences in natural language because saying those sentences helps us get by in a complex society. 
We need learning not just for erudition, but because having a better idea of how the world works 
enables us to generate more effective strategies for dealing with it. We need visual perception not 
just because seeing is fun, but in order to get a better idea of what an action might achieve — for 
example, being able to see a tasty morsel helps one to move toward it. 

The study of AI as rational agent design therefore has two advantages. First, it is more 
general than the "laws of thought" approach, because correct inference is only a useful mechanism 
for achieving rationality, and not a necessary one. Second, it is more amenable to scientific 
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development than approaches based on human behavior or human thought, because the standard 
of rationality is clearly defined and completely general. Human behavior, on the other hand, 
is well-adapted for one specific environment and is the product, in part, of a complicated and 
largely unknown evolutionary process that still may be far from achieving perfection. This 
book will therefore concentrate on general principles of rational agents, and on componentsfor 
constructing them. We will see that despite the apparent simplicity with which the problem can 
be stated, an enormous variety of issues come up when we try to solve it. Chapter 2 outlines 
some of these issues in more detail. 

One important point to keep in mind: we will see before too long that achieving perfect 
rationality — always doing the right thing — is not possible in complicated environments. The 
computational demands are just too high. However, for most of the book, we will adopt the 
working hypothesis that understanding perfect decision making is a good place to start. It 
simplifies the problem and provides the appropriate setting for most of the foundational material 
in the field. Chapters 5 and 17 deal explicitly with the issue of limited rationality — acting 
appropriately when there is not enough time to do all the computations one might like. 



1 ,2 THE FOUNDATIONS OF ARTIFICIAL INTELLIGENCE 



In this section and the next, we provide a brief history of AI. Although AI itself is a young field, 
it has inherited many ideas, viewpoints, and techniques from other disciplines. From over 2000 
years of tradition in philosophy, theories of reasoning and learning have emerged, along with the 
viewpoint that the mind is constituted by the operation of a physical system. From over 400 years 
of mathematics, we have formal theories of logic, probability, decision making, and computation. 
From psychology, we have the tools with which to investigate the human mind, and a scientific 
language within which to express the resulting theories. From linguistics, we have theories of 
the structure and meaning of language. Finally, from computer science, we have the tools with 
which to make AI a reality. 

Like any history, this one is forced to concentrate on a small number of people and events, 
and ignore others that were also important. We choose to arrange events to tell the story of how 
the various intellectual components of modem AI came into being. We certainly would not wish 
to give the impression, however, that the disciplines from which the components came have all 
been working toward AI as their ultimate fruition. 



Philosophy (428 B.C.-present) 

The safest characterization of the European philosophical tradition is that it consists of a series 
of footnotes to Plato. 

— Alfred North Whitehead 

We begin with the birth of Plato in 428 B.C. His writings range across politics, mathematics, 
physics, astronomy, and several branches of philosophy. Together, Plato, his teacher Socrates, 
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and his student Aristotle laid the foundation for much of western thought and culture. The 
philosopher Hubert Dreyfus (1979, p. 67) says that "The story of artificial intelligence might well 
begin around 450 B.C." when Plato reported a dialogue in which Socrates asks Euthyphro, 3 "I 
want to know what is characteristic of piety which makes all actions pious . . . that I may have it 
to turn to, and to use as a standard whereby to judge your actions and those of other men.” 4 In 
other words, Socrates was asking for an algorithm to distinguish piety from non-piety. Aristotle 
went on to try to formulate more precisely the laws governing the rational part of the mind. He 
developed an informal system of syllogisms for proper reasoning, which in principle allowed one 
to mechanically generate conclusions, given initial premises. Aristotle did not believe all parts 
of the mind were governed by logical processes; he also had a notion of intuitive reason. 

Now that we have the idea of a set of rules that can describe the working of (at least part 
of) the mind, the next step is to consider the mind as a physical system. We have to wait for 
Rene Descartes (1596-1650) for a clear discussion of the distinction between mind and matter, 
and the problems that arise. One problem with a purely physical conception of the mind is that 
it seems to leave little room for free will: if the mind is governed entirely by physical laws, then 
it has no more free will than a rock "deciding" to fall toward the center of the earth. Although a 
dualism strong advocate of the power of reasoning, Descartes was also a proponent of dualism. He held 

that there is a part of the mind (or soul or spirit) that is outside of nature, exempt from physical 
laws. On the other hand, he felt that animals did not possess this dualist quality; they could be 
considered as if they were machines. 

materialism An alternative to dualism is materialism, which holds that all the world (including the 

brain and mind) operate according to physical law. 5 Wilhelm Leibniz (1646-1716) was probably 
the first to take the materialist position to its logical conclusion and build a mechanical device 
intended to carry out mental operations. Unfortunately, his formulation of logic was so weak that 
his mechanical concept generator could not produce interesting results. 

It is also possible to adopt an intermediate position, in which one accepts that the mind 
has a physical basis, but denies that it can be explained by a reduction to ordinary physical 
processes. Mental processes and consciousness are therefore part of the physical world, but 
inherently unknowable; they are beyond rational understanding. Some philosophers critical of 
AI have adopted exactly this position, as we discuss in Chapter 26. 

Barring these possible objections to the aims of AI, philosophy had thus established a 
tradition in which the mind was conceived of as a physical device operating principally by 
reasoning with the knowledge that it contained. The next problem is then to establish the 
empiricist source of knowledge. The empiricist movement, starting with Francis Bacon's (1561-1626) 

Novum Organum, 6 is characterized by the dictum of John Locke (1632-1704): "Nothing is in 
the understanding, which was not first in the senses." David Hume's (17 11-1776) A Treatise 
induction of Human Nature (Hume, 1978) proposed what is now known as the principle of induction: 



3 The Euthyphro describes the events just before the trial of Socrates in 399 B.C. Dreyfus has clearly erred in placing it 
5 1 years earlier. 

4 Note that other translations have "goodness/good" instead of "piety/pious." 

5 In this view, the perception of "free will" arises because the deterministic generation of behavior is constituted by the 
operation of the mind selecting among what appear to be the possible courses of action. They remain possible because 
the brain does not have access to its own future states. 

6 An update of Aristotle's organon, or instrument of thought. 
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that general rules are acquired by exposure to repeated associations between their elements. 
The theory was given more formal shape by Bertrand Russell (1872-1970) who introduced 
logical positivism logical positivism. This doctrine holds that all knowledge can be characterized by logical 
sentence™ theories connected, ultimately, to observation sentences that correspond to sensory inputs. 7 The 

theory MATION confirmation theory of Rudolf Carnap and Carl Hempel attempted to establish the nature of the 

connection between the observation sentences and the more general theories — in other words, to 
understand how knowledge can be acquired from experience. 

The final element in the philosophical picture of the mind is the connection between 
knowledge and action. What form should this connection take, and how can particular actions 
be justified? These questions are vital to AI. because only by understanding how actions are 
justified can we understand how to build an agent whose actions are justifiable, or rational. 
Aristotle provides an elegant answer in the Nicomachean Ethics (Book III. 3, 1 1 12b): 



We deliberate not about ends,, but about means. For a doctor does not deliberate whether he 
shall heal, nor an orator whether he shall persuade, nor a statesman whether he shall produce 
law and order, nor does any one else deliberate about his end. They assume the end and 
consider how and by what means it is attained, and if it seems easily and best produced 
thereby; while if it is achieved by one means only they consider how it will be achieved by 
this and by what means this will be achieved, till they come to the first cause, which in the 
order of discovery is last . . . and what is last in the order of analysis seems to be first in the 
order of becoming. And if we come on an impossibility, we give up the search, e.g. if we 
need money and this cannot be got: but if a thing appears possible we try to do it. 

Aristotle's approach (with a few minor refinements) was implemented 2300 years later by Newell 
and Simon in their GPS program, about which they write (Newell and Simon, 1972): 

The main methods of GPS jointly embody the heuristic of means-ends analysis. Means-ends 
analysis is typified by the following kind of common-sense argument: 

I want to take my son to nursery school. What's the difference between what I 
have and what I want? One of distance. What changes distance? My automobile. 

My automobile won't work. What is needed to make it work? A new battery. 

What has new batteries? An auto repair shop. I want the repair shop to put in a 
new battery; but the shop doesn't know I need one. What is the difficulty? One 
of communication. What allows communication? A telephone . . . and so on. 

This kind of analysis — classifying things in terms of the functions they serve and oscillating 
among ends, functions required, and means that perform them — forms the basic system of 
heuristic of GPS. 

Means-ends analysis is useful, but does not say what to do when several actions will achieve the 
goal, or when no action will completely achieve it. Arnauld, a follower of Descartes, correctly 
described a quantitative formula for deciding what action to take in cases like this (see Chapter 16). 
John Stuart Mill's ( 1 806-1 873)book Utilitarianism (Mill, 1 863 ) amplifies on this idea. The more 
formal theory of decisions is discussed in the following section. 



7 In this picture, all meaningful statements can be verified or falsified either by analyzing the meaning of the words or 
by carrying out experiments. Because this rules out most of metaphysics, as was the intention, logical positivism was 
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Mathematics (c. 800-present) 

Philosophers staked out most of the important ideas of AI, but to make the leap to a formal 
science required a level of mathematical formalization in three main areas: computation, logic, 
ALGORITHM and probability. The notion of expressing a computation as a formal algorithm goes back to 

al-Khowarazmi, an Arab mathematician of the ninth century, whose writings also introduced 
Europe to Arabic numerals and algebra. 

Logic goes back at least to Aristotle, but it was a philosophical rather than mathematical 
subject until George Boole (1815-1864) introduced his formal language for making logical 
inference in 1847. Boole's approach was incomplete, but good enough that others filled in the 
gaps. In 1879, Gottlob Frege (1848-1925) produced a logic that, except for some notational 
changes, forms the first-order logic that is used today as the most basic knowledge representation 
system. 8 Alfred Tarski (1902-1983) introduced a theory of reference that shows how to relate 
the objects in a logic to objects in the real world. The next step was to determine the limits of 
what could be done with logic and computation. 

David Hilbert (1862-1943), a great mathematician in his own right, is most remembered 
for the problems he did not solve. In 1 900, he presented a list of 23 problems that he correctly 
predicted would occupy mathematicians for the bulk of the century. The final problem asks 
if there is an algorithm for deciding the truth of any logical proposition involving the natural 
numbers — the famous Entscheidungsproblem , or decision problem. Essentially, Hilbert was 
asking if there were fundamental limits to the power of effective proof procedures. In 1930, Kurt 
Godel (1906-1978) showed that there exists an effective procedure to prove any true statement in 
the first-order logic of Frege and Russell; but first-order logic could not capture the principle of 
mathematical induction needed to characterize the natural numbers. In 1931, he showed that real 
m C E0RE L M ETENESS limits do exist. His incompleteness theorem showed that in any language expressive enough 
to describe the properties of the natural numbers, there are true statements that are undecidable: 
their truth cannot be established by any algorithm. 

This fundamental result can also be interpreted as showing that there are some functions 
on the integers that cannot be represented by an algorithm — that is, they cannot be computed. 
This motivated Alan Turing (1912-1954) to try to characterize exactly which functions are 
capable of being computed. This notion is actually slightly problematic, because the notion 
of a computation or effective procedure really cannot be given a formal definition. However, 
the Church-Turing thesis, which states that the Turing machine (Turing, 1936) is capable of 
computing any computable function, is generally accepted as providing a sufficient definition. 
Turing also showed that there were some functions that no Turing machine can compute. For 
example, no machine can tell in general whether a given program will return an answer on a 
given input, or run forever. 

Although undecidability and noncomputability are important to an understanding of com- 
wtractability putation, the notion of intractability has had a much greater impact. Roughly speaking, 
a class of problems is called intractable if the time required to solve instances of the class 
grows at least exponentially with the size of the instances. The distinction between polynomial 
and exponential growth in complexity was first emphasized in the mid-1960s (Cobham, 1964; 
Edmonds, 1965). It is important because exponential growth means that even moderate-sized in- 

8 To understand why Frege's notation was not universally adopted, see the cover of this book. 
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stances cannot be solved in any reasonable time. Therefore, one should strive to divide the overall 
problem of generating intelligent behavior into tractable subproblems rather than intractable ones. 

REDUCTION The second important concept in the theory of complexity is reduction, which also emerged in 

the 1960s (Dantzig, 1960; Edmonds, 1962). A reduction is a general transformation from one 
class of problems to another, such that solutions to the first class can be found by reducing them 
to problems of the second class and solving the latter problems. 

NP COMPLETENESS How can one recognize an intractable problem? The theory ofNP-completeness, pioneered 

by Steven Cook (1971) and Richard Karp (1972), provides a method. Cook and Karp showed 
the existence of large classes of canonical combinatorial search and reasoning problems that 
are NP-complete. Any problem class to which an NP-complete problem class can be reduced 
is likely to be intractable. (Although it has not yet been proved that NP-complete problems 
are necessarily intractable, few theoreticians believe otherwise.) These results contrast sharply 
with the "Electronic Super-Brain" enthusiasm accompanying the advent of computers. Despite 
the ever-increasing speed of computers, subtlety and careful use of resources will characterize 
intelligent systems. Put crudely, the world is an extremely large problem instance! 

Besides logic and computation, the third great contribution of mathematics to AI is the j 
theory of probability. The Italian Gerolamo Cardano (1501-1576) first framed the idea of! 
probability, describing it in terms of the possible outcomes of gambling events. Before his time, ; 
the outcomes of gambling games were seen as the will of the gods ratherthan the whim of chance, i 
Probability quickly became an invaluable part of all the quantitative sciences, helping to deal 
with uncertain measurements and incomplete theories. Pierre Fermat ( 1 601-1665), Blaise Pascal 1 
(1623-1662), James Bernoulli (1654-1705), Pierre Laplace (1749-1827), and others advanced 
the theory and introduced new statistical methods. Bernoulli also framed an alternative view 
of probability, as a subjective "degree of belief" rather than an objective ratio of outcomes. 
Subjective probabilities therefore can be updated as new evidence is obtained. Thomas Bayes 
(1702-1761) proposed a rule for updating subjective probabilities in the light of new evidence 
(published posthumously in 1763). Bayes' rule, and the subsequent field of Bayesian analysis, 
form the basis of the modern approach to uncertain reasoning in AI systems. Debate still rages 
between supporters of the objective and subjective views of probability, but it is not clear if thei 
difference has great significance for AI. Both versions obey the same set of axioms. Savage's] 
(1954) Foundations of Statistics gives a good introduction to the field. 

As with logic, a connection must be made between probabilistic reasoning and action. 

DECISION THEORY Decision theory, pioneered by John Von Neumann and Oskar Morgenstem (1944), combines 
probability theory with utility theory (which provides a formal and complete framework for 
specifying the preferences of an agent) to give the first general theory that can distinguish good 
actions from bad ones. Decision theory is the mathematical successor to utilitarianism, and 
provides the theoretical basis for many of the agent designs in this book. 



Psychology (1879-present) 

Scientific psychology can be said to have begun with the work of the German physicist Hermann 
von Helmholtz (1821-1894) and his student Wilhelm Wundt (1832-1920). Helmholtz applied 
the scientific method to the study of human vision, and his Handbook of Physiological Optics 
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is even now described as "the single most important treatise on the physics and physiology of 
human vision to this day" (Nalwa, 1993, p.15). In 1879, the same year that Frege launched first- 
order logic, Wundt opened the first laboratory of experimental psychology at the University of 
Leipzig. Wundt insisted on carefully controlled experiments in which his workers would perform 
a perceptual or associative task while introspecting on their thought processes. The careful 
controls went a long way to make psychology a science, but as the methodology spread, a curious 
phenomenon arose: each laboratory would report introspective data that just happened to match 
behaviorism the theories tint were popular in that laboratory. The behaviorism movement of John Watson 

(1878-1958) aid Edward Lee Thorndike (1874-1949) rebelled against this subj ectivism, rej ecting 
any theory involving mental processes on the grounds that introspection could not provide reliable 
evidence. Behiviorists insisted on studying only objective measures of the percepts (or stimulus) 
given to an animal and its resulting actions (or response). Mental constructs such as knowledge, 
beliefs, goals, ind reasoning steps were dismissed as unscientific “folk psychology.” Behaviorism 
discovered a let about rats and pigeons, but had less success understanding humans. Nevertheless, 
it had a stronghold on psychology (especially in the United States) from about 1920 to 1960. 
The view that the brain possesses and processes information, which is the principal char- 
psychology acteristic of cognitive psychology, can be traced back at least to the works of William James 9 
(1842-1910). Helmholtz also insisted that perception involved a form of unconscious logical in- 
ference. The cognitive viewpoint was largely eclipsed by behaviorism until 1943, when Kenneth 
Craik published The Nature of Explanation. Craik put back the missing mental step between 
stimulus and response. He claimed that beliefs, goals, and reasoning steps could be useful valid 
components of a theory of human behavior, and are just as scientific as, say, using pressure and 
temperature to talk about gases, despite their being made of molecules that have neither. Craik 
specified the three key steps of a knowledge-based agent: (1) the stimulus must be translated into 
an internal representation, (2) the representation is manipulated by cognitive processes to derive 
new internal representations, and (3) these are in turn retranslated back into action. He clearly 
explained why this was a good design for an agent: 

If the orginism carries a "small-scale model" of external reality and of its own possible actions 
within itf head, it is able to try out various alternatives, conclude which is the best of them, 
react to fiture situations before they arise, utilize the knowledge of past events in dealing with 
the present and future, and in every way to react in a much fuller, safer, and more competent 
manner to the emergencies which face it. (Craik, 1943) 

An agent designed this way can, for example, plan a long trip by considering various possi- 
ble routes, comparing them, and choosing the best one, all before starting the journey. Since 
the 1960s, the information-processing view has dominated psychology. It it now almost taken 
for granted among many psychologists that "a cognitive theory should be like a computer pro- 
gram" (Anderson, 1 980). By this it is meant that the theory should describe cognition as consisting 
of well-definei transformation processes operating at the level of the information carried by the 
input signals. 

For most of the early history of AI and cognitive science, no significant distinction was 
drawn between the two fields, and it was common to see AI programs described as psychological 

9 William James was the brother of novelist Henry James. It is said that Henry wrote fiction as if it were psychology 
and William wrote psychology as if it were fiction. 
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results without any claim as to the exact human behavior they were modelling. In the last decade 
or so, however, the methodological distinctions have become clearer, and most work now falls 
into one field or the other. 



Computer engineering (1940-present) 

For artificial intelligence to succeed, we need two things: intelligence and an artifact. The 
computer has been unanimously acclaimed as the artifact with the best chance of demonstrating 
intelligence. The modern digital electronic computer was invented independently and almost 
simultaneously by scientists in three countries embattled in World War II. The first operational 
modern computer was the Heath Robinson, 10 built in 1940 by Alan Turing's team for the single 
purpose of deciphering German messages. When the Germans switched to a more sophisticated 
code, the electromechanical relays in the Robinson proved to be too slow, and a new machine 
called the Colossus was built from vacuum tubes. It was completed in 1943, and by the end of 
the war, ten Colossus machines were in everyday use. 

The first operational programmable computer was the Z-3, the invention of Konrad Zuse 
in Germany in 1941. Zuse invented floating-point numbers for the Zr3, and went on in 1945 to 
develop Plankalkul, the first high-level programming language. Although Zuse received some 
support from the Third Reich to apply his machine to aircraft design, the military hierarchy did 
not attach as much importance to computing as did its counterpart in Britain. 

In the United States, the first electronic computer, the ABC, was assembled by John 
Atanasoff and his graduate student Clifford Berry between 1940and 1942 at Iowa State University. 
The project received little support and was abandoned after Atanasoff became involved in military 
research in Washington. Two other computer projects were started as secret military research: 
the Mark I, II, and III computers were developed at Harvard by a team under Howard Aiken; and 
the ENIAC was developed at the University of Pennsylvania by a team including John Mauchly 
and John Eckert. ENIAC was the first general-purpose, electronic, digital computer. One of its 
first applications was computing artillery firing tables. A successor, the EDVAC, followed John 
Von Neumann's suggestion to use a stored program, so that technicians would not have to scurry 
about changing patch cords to run a new program. 

But perhaps the most critical breakthrough was the IBM 701, built in 1952 by Nathaniel 
Rochester and his group. This was the first computer to yield a profit for its manufacturer. IBM 
went on to become one of the world’s largest corporations, and sales of computers have grown to j 
$150 billion/year. In the United States, the computer industry (including software and services) j 
now accounts for about 10% of the gross national product. 

Each generation of computer hardware has brought an increase in speed and capacity, and I 
a decrease in price. Computer engineering has been remarkably successful, regularly doubling 
performance every two years, with no immediate end in sight for this rate of increase. Massively j 
parallel machines promise to add several more zeros to the overall throughput achievable. 

Of course, there were calculating devices before the electronic computer. The abacus j 
is roughly 7000 years old. In the mid-17th century, Blaise Pascal built a mechanical adding 1 



10 Heath Robinson was a cartoonist famous for his depictions of whimsical and absurdly complicated contraptions for 
everyday tasks such as buttering toast. 
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and subtracting machine called the Pascaline. Leibniz improved on this in 1694. building a 
mechanical device that multiplied by doing repeated addition. Progress stalled for over a century 
until Charles Babbage ( 1792-1871 (dreamed that logarithm tables could be computed by machine. 
He designed a machine for this task, but never completed the project. Instead, he turned to the 
design of the Analytical Engine, for which Babbage invented the ideas of addressable memory, 
stored programs, and conditional jumps. Although the idea of programmable machines was 
not new — in 1805. Joseph Marie Jacquard invented a loom that could be programmed using 
punched cards — Babbage's machine was the first artifact possessing the characteristics necessary 
for universal computation. Babbage's colleague Ada Lovelace, daughter of the poet Lord Byron, 
wrote programs forthe Analytical Engine and even speculated that the machine could play chess 
or compose music. Lovelace was the world's first programmer, and the first of many to endure 
massive cost overruns and to have an ambitious project ultimately abandoned. 1 1 Babbage's basic 
design was proven viable by Doron Swade and his colleagues, who built a working model using 
only the mechanical techniques available at Babbage's time (Swade. 1993). Babbage had the 
right idea, but lacked the organizational skills to get his machine built. 

AI also owes a debt to the software side of computer science, which has supplied the 
operating systems, programming languages, and tools needed to write modern programs (and 
papers about them). But this is one area where the debt has been repaid: work in AI has pioneered 
many ideas that have made their way back to "mainstream" computer science, including time 
sharing, interactive interpreters, the linked list data type, automatic storage management, and 
some of the key concepts of object-oriented programming and integrated program development 
environments with graphical user interfaces. 

Linguistics (1957-present) 

In 1957. B. F. Skinner published Verbal Behavior. This was a comprehensive, detailed account 
of the behaviorist approach to language learning, written by the foremost expert in the field. But 
curiously, a review of the book became as well-known as the book itself, and served to almost kill 
off interest in behaviorism. The authorofthe review was Noam Chomsky, who had just published 
a book on his own theory. Syntactic Structures. Chomsky showed how the behaviorist theory did 
not address the notion of creativity in language — it did not explain how a child could understand 
and make up sentences that he or she had never heard before. Chomsky's theory — based on 
syntactic models going back to the Indian linguist Panini (c. 350 B.C.) — could explain this, and 
unlike previous theories, it was formal enough that it could in principle be programmed. 

Later developments in linguistics showed the problem to be considerably more complex 
than it seemed in 1957. Language is ambiguous and leaves much unsaid. This means that 
understanding language requires an understanding of the subject matter and context, not just an 
understanding of the structure of sentences. This may seem obvious, but it was not appreciated 
until the early 1960s. Much of the early work in knowledge representation (the study of how to 
put knowledge into a form that a computer can reason with) was tied to language and informed 
by research in linguistics, which was connected in turn to decades of work on the philosophical 
analysis of language. 



She also gave her : 



Ada. the U.S. Department of 1 
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Modem linguistics and AI were "bom" at about the same time, so linguistics does not play 
a large foundational role in the growth of AI. Instead, the two grew up together, intersecting 
in a hybrid field called computational linguistics or natural language processing, which 
concentrates on the problem of language use. 



1 ,3 THE HISTORY OF ARTIFICIAL INTELLIGENCE 



With the background material behind us, we are now ready to outline the development of AI 
proper. We could do this by identifying loosely defined and overlapping phases in its development, 
or by chronicling the various different and intertwined conceptual threads that make up the field. 
In this section, we will take the former approach, at the risk of doing some degree of violence 
to the real relationships among subfields. The history of each subfield is covered in individual 
chapters later in the book. 



The gestation of artificial intelligence (1943-1956) 

The first work that is now generally recognized as AI was done by Warren McCulloch and 
Walter Pitts (1943). They drew on three sources: knowledge of the basic physiology and 
function of neurons in the brain; the formal analysis of propositional logic due to Russell and 
Whitehead; and Turing's theory of computation. They proposed a model of artificial neurons in 
which each neuron is characterized as being "on" or "off," with a switch to "on" occurring in 
response to stimulation by a sufficient number of neighboring neurons. The state of a neuron 
was conceived of as "factually equivalent to a proposition which proposed its adequate stimulus." 
They showed, for example, that any computable function could be computed by some network 
of connected neurons, and that all the logical connectives could be implemented by simple 
net structures. McCulloch and Pitts also suggested that suitably defined networks could learn. 
Donald Hebb ( 1 949) demonstrated a simple updating rule for modifying the connection strengths 
between neurons, such that learning could take place. 

The work of McCulloch and Pitts was arguably the forerunner of both the logicist tradition 
in AI and the connectionist tradition. In the early 1950s, Claude Shannon (1950) and Alan 
Turing (1953) were writing chess programs for von Neumann-style conventional computers. 12 
At the same time, two graduate students in the Princeton mathematics department, Marvin 
Minsky and Dean Edmonds, built the first neural network computer in 1951. The SNARC, as 
it was called, used 3000 vacuum tubes and a surplus automatic pilot mechanism from a B-24 
bomber to simulate a network of 40 neurons. Minsky's Ph.D. committee was skeptical whether 
this kind of work should be considered mathematics, but von Neumann was on the committee 
and reportedly said, "If it isn't now it will be someday.” Ironically, Minsky was later to prove 
theorems that contributed to the demise of much of neural network research during the 1 970s. 



12 Shannon actually had no real computer to work with, and Turing was eventually denied access to his own team's 
computers by the British government, on the grounds that research into artificial intelligence was surely frivolous. 
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Princeton was home to another influential figure in AI, John McCarthy. After graduation, 
McCarthy moved to Dartmouth College, which was to become the official birthplace of the 
field. McCarthy convinced Minsky, Claude Shannon, and Nathaniel Rochester to help him bring 
together U.S. researchers interested in automata theory, neural nets, and the study of intelligence. 
They organized a two-month workshop at Dartmouth in the summer of 1956. All together there 
were ten attendees, including Trenchard More from Princeton, Arthur Samuel from IBM, and 
Ray Solomonoff and Oliver Selfridge from MIT. 

Two researchers from Carnegie Tech, 13 Alien Newell and Herbert Simon, rather stole the 
show. Although the others had ideas and in some cases programs for particular applications 
such as checkers, Newell and Simon already had a reasoning program, the Logic Theorist (LT), 
about which Simon claimed, "We have invented a computer program capable of thinking non- 
numerically, and thereby solved the venerable mind-body problem.” 14 Soon after the workshop, 
the program was able to prove most of the theorems in Chapter 2 of Russell and Whitehead’s 
Principia Mathematica. Russell was reportedly delighted when Simon showed him that the pro- 
gram had come up with a proof for one theorem that was shorter than the one in Principia. The 
editors of the Journal of Symbolic Logic were less impressed; they rejected a paper coauthored 
by Newell, Simon, and Logic Theorist. 

The Dartmouth workshop did not lead to any new breakthroughs, but it did introduce all 
the major figures to each other. For the next 20 years, the field would be dominated by these 
people and their students and colleagues at MIT, CMU, Stanford, and IBM. Perhaps the most 
lasting thing to come out of the workshop was an agreement to adopt McCarthy's new name for 
the field: artificial intelligence. 



Early enthusiasm, great expectations (1952-1969) 

The early years of AI were full of successes — in a limited way. Given the primitive computers 
and programming tools of the time, and the fact that only a few years earlier computers were 
seen as things that could do arithmetic and no more, it was astonishing whenever a computer did 
anything remotely clever. The intellectual establishment, by and large, preferred to believe that "a 
machine can never do X" (see Chapter 26 for a long list ofX'v gathered by Turing). AI researchers 
naturally responded by demonstrating one X after another. Some modem AI researchers refer to 
this period as the "Look, Ma, no hands!" era. 

Newell and Simon's early success was followed up with the General Problem Solver, 
or GPS. Unlike Logic Theorist, this program was designed from the start to imitate human 
problem-solving protocols. Within the limited class of puzzles it could handle, it turned out that 
the order in which the program considered subgoals and possible actions was similar to the way 
humans approached the same problems. Thus, GPS was probably the first program to embody 
the "thinking humanly" approach. The combination of AI and cognitive science has continued 
at CMU up to the present day. 



13 Now Carnegie Mellon University (CMU). 

14 Newell and Simon also invented a list-processing language, IPL, to write LT. They had no compiler, and translated it 
into machine code by hand. To avoid errors, they worked in parallel, calling out binary numbers to each other as they 
wrote each instruction to make sure they agreed. 
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At IBM, Nathaniel Rochester and his colleagues produced some of the first AI programs. 
Herbert Gelemter (1959) constructed the Geometry Theorem Prover. Like the Logic Theorist, 
it proved theorems using explicitly represented axioms. Gelernter soon found that there were 
too many possible reasoning paths to follow, most of which turned out to be dead ends. To help 
focus the search, he added the capability to create a numerical representation of a diagram — a 
particular case of the general theorem to be proved. Before the program tried to prove something, 
it could first check the diagram to see if it was true in the particular case. 

Starting in 1952, Arthur Samuel wrote a series of programs for checkers (draughts) that 
eventually learned to play toumament-level checkers. Along the way, he disproved the idea that 
computers can only do what they are told to, as his program quickly learned to play a better game 
than its creator. The program was demonstrated on television in February 1956, creating a very 
strong impression. Like Turing, Samuel had trouble finding computer time. Working at night, he 
used machines that were still on the testing floor at IBM's manufacturing plant. Chapter 5 covers 
game playing, and Chapter 20 describes and expands on the learning techniques used by Samuel. 

John McCarthy moved from Dartmouth to MIT and there made three crucial contributions 
in one historic year: 1958. In MIT AI Lab Memo No. 1, McCarthy defined the high-level language 
•DSP Lisp, which was to become the dominant AI programming language. Lisp is the second-oldest 

language in current use. 15 With Lisp, McCarthy had the tool he needed, but access to scarce and 
expensive computing resources was also a serious problem. Thus, he and others at MIT invented 
time sharing. After getting an experimental time-sharing system up at MIT, McCarthy eventually 
attracted the interest of a group of MIT grads who formed Digital Equipment Corporation, which 
was to become the world's second largest computer manufacturer, thanks to their time-sharing 
minicomputers. Also in 1958, McCarthy published a paper entitled Programs with Common 
Sense, in which he described the Advice Taker, a hypothetical program that can be seen as the 
first complete AI system. Like the Logic Theorist and Geometry Theorem Prover, McCarthy's 
program was designed to use knowledge to search for solutions to problems. B ut unlike the others, 
it was to embody general knowledge of the world. For example, he showed how some simple 
axioms would enable the program to generate a plan to drive to the airport to catch a plane. The 
program was also designed so that it could accept new axioms in the normal course of operation, 
thereby allowing it to achieve competence in new areas without being reprogrammed. The Advice 
Taker thus embodied the central principles of knowledge representation and reasoning: that it 
is useful to have a formal, explicit representation of the world and the way an agent's actions 
affect the world, and to be able to manipulate these representations with deductive processes. It 
is remarkable how much of the 1958 paper remains relevant after more than 35 years. 

1958 also marked the year that Marvin Minsky moved to MIT. For years he and McCarthy 
were inseparable as they defined the field together. But they grew apart as McCarthy stressed 
representation and reasoning in formal logic, whereas Minsky was more interested in getting 
programs to work, and eventually developed an anti-logical outlook. In 1963, McCarthy took 
the opportunity to go to Stanford and start the AI lab there. His research agenda of using 
logic to build the ultimate Advice Taker was advanced by J. A. Robinson's discovery of the 
resolution method (a complete theorem-proving algorithm for first-order logic; see Section 9.6). 
Work at Stanford emphasized general-purpose methods for logical reasoning. Applications of 



15 FORTRAN is one year older than Lisp. 
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logic included Cordell Green's question answering and planning systems (Green, 1969b), and the 
Shakey robotics project at the new Stanford Research Institute (SRI). The latter project, discussed 
further in Chapter 25, was the first to demonstrate the complete integration of logical reasoning 
and physical activity. 

Minsky supervised a series of students who chose limited problems that appeared to require 
microworlds intelligence to solve. These limited domains became known as microworlds. James Slagle's 
SAINT program (1963a) was able to solve closed-form integration problems typical of first-year 
college calculus courses. Tom Evans’s ANALOGY program (1968) solved geometric analogy 
problems that appear in IQ tests, such as the one in Figure 1 .2. Bertram Raphael's (1968) SIR 
(Semantic Information Retrieval) was able to accept input statements in a very restricted subset 
of English and answer questions thereon. Daniel Bobrow's STUDENT program (1967) solved 
algebra story problems such as 

If the number of customers Tom gets is twice the square of 20 percent of the number of 
advertisements he runs, and the number of advertisements he runs is 45, what is the number 
of customers Tom gets? 




The most famous micro world was the blocks world, which consists of a set of solid blocks 
placed on a tabletop (or more often, a simulation of a tabletop), as shown in Figure 1.3. A task 
in this world is to rearrange the blocks in a certain way, using a robot hand that can pick up one 
block at a time. The blocks world was home to the vision project of David Huffman (1971), 
the vision and constraint-propagation work of David Waltz (1975), the learning theory of Patrick 
Winston (1970), the natural language understanding program of Terry Winograd (1972), and the 
planner of Scott Fahlman (1974). 

Early work building on the neural networks of McCulloch and Pitts also flourished. The 
work of Winograd and Cowan (1963) showed how a large number of elements could collectively 
represent an individual concept, with a corresponding increase in robustness and parallelism. 
Hebb's learning methods were enhanced by Bernie Widrow (Widrow and Hoff, 1960; Widrow, 
1962), who called his networks adalines, and by Frank Rosenblatt (1962) with his perceptrons. 
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Figure 1.3 A scene from the blocks world. A task for the robot might be "Pick up a big red 
block," expressed either in natural language or in a formal notation. 



Rosenblatt proved the famous perceptron convergence theorem, showing that his learning 
algorithm could adjust the connection strengths of a perceptron to match any input data, provided 
such a match existed. These topics are covered in Section 19.3. 



A dose of reality (1966-1974) 

From the beginning, AI researchers were not shy in making predictions of their coming successes. 
The following statement by Herbert Simon in 1 957 is often quoted: 

It is not my aim to surprise or shock you — but the simplest way I can summarize is to say 
that there are now in the world machines that think, that learn and that create. Moreover, their 
ability to do these things is going to increase rapidly until — in a visible future — the range of 
problems they can handle will be coextensive with the range to which human mind has been 
applied. 

Although one might argue that terms such as "visible future" can be interpreted in various ways, 
some of Simon's predictions were more concrete. In 1958, he predicted that within 10 years 
a computer would be chess champion, and an important new mathematical theorem would be 
proved by machine. Claims such as these turned out to be wildly optimistic. The barrier that 
faced almost all AI research projects was that methods that sufficed for demonstrations on one or 
two simple examples turned out to fail miserably when tried out on wider selections of problems 
and on more difficult problems. 

The first kind of difficulty arose because early programs often contained little or no 
knowledge of their subject matter, and succeeded by means of simple syntactic manipulations. 
Weizenbaum's ELIZA program (1965), which could apparently engage in serious conversation 
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on any topic, actually just borrowed and manipulated the sentences typed into it by a human. 
A typical stray occurred in early machine translation efforts, which were generously funded by 
the National Research Council in an attempt to speed up the translation of Russian scientific 
papers in the wake of the Sputnik launch in 1957. It was thought initially that simple syntactic 
transformations based on the grammars of Russian and English, and word replacement using 
an electronic dictionary, would suffice to preserve the exact meanings of sentences. In fact, 
translation requires general knowledge of the subject matter in order to resolve ambiguity and 
establish the content of the sentence. The famous retranslation of "the spirit is willing but the 
flesh is weak' as "the vodka is good but the meat is rotten" illustrates the difficulties encountered. 
In 1 966, a report by an advisory committee found that "there has been no machine translation 
of general scientific text, and none is in immediate prospect." All U.S. government funding for 
academic translation projects was cancelled. 

The second kind of difficulty was the intractability of many of the problems that AI was 
attempting tosolve. Most of the early AI programs worked by representing the basic facts about 
a problem and trying out a series of steps to solve it, combining different combinations of steps 
until the right one was found. The early programs were feasible only because microworlds 
contained very few objects. Before the theory of NP-completeness was developed, it was widely 
thought that "scaling up" to larger problems was simply a matter of faster hardware and larger 
memories. The optimism that accompanied the development of resolution theorem proving, for 
example, was soon dampened when researchers failed to prove theorems involving more than a 
. few dozen fans. The fact that a program can find a solution in principle does not mean that the 

■ program contains any of the mechanisms needed to find it in practice. 

The illusion of unlimited computational power was not confined to problem-solving pro- 
machine evolution grams. Earh experiments in machine evolution (now called genetic algorithms) (Friedberg, 
1958; Friedberg et al. , 1959) were based on the undoubtedly correct belief that by making an 
appropriate series of small mutations to a machine code program, one can generate a program 
with good performance for any particular simple task. The idea, then, was to try random muta- 
tions and then apply a selection process to preserve mutations that seemed to improve behavior. 
Despite thousands of hours of CPU time, almost no progress was demonstrated. 

Failure to come to grips with the "combinatorial explosion" was one of the main criticisms 
of AI contained in the Lighthill report (Lighthill, 1973), which formed the basis for the decision 
by the British government to end support for AI research in all but two universities. (Oral 
tradition painls a somewhat different and more colorful picture, with political ambitions and 
personal animosities that cannot be put in print.) 

A third difficulty arose because of some fundamental limitations on the basic structures 
being used to generate intelligent behavior. For example, in 1969, Minsky and Papert's book 
Perceptrons (1969) proved that although perceptrons could be shown to leam anything they were 
capable of representing, they could represent very little. In particular, a two-input perceptron 
could not be :rained to recognize when its two inputs were different. Although their results 
did not apply to more complex, multilayer networks, research funding for neural net research 
soon dwindled to almost nothing. Ironically, the new back-propagation learning algorithms for 
multilayer networks that were to cause an enormous resurgence in neural net research in the late 
1980s were actually discovered first in 1969 (Bryson and Ho, 1969). 



Chapter 1 . Introduction 



Knowledge-based systems: The key to power? (1969-1979) 

The picture of problem solving that had arisen during the first decade of AI research was of a 
general-purpose search mechanism trying to string together elementary reasoning steps to find 
weak methods complete solutions. Such approaches have been called weak methods, because they use weak 
information about the domain. For many complex domains, it turns out that their performance is 
also weak. The only way around this is to use knowledge more suited to making larger reasoning 
steps and to solving typically occurring cases in narrow areas of expertise. One might say that to 
solve a hard problem, you almost have to know the answer already. 

The DENDRAL program (Buchanan et al., 1969) was an early example of this approach. It 
was developed at Stanford, where Ed Feigenbaum (a former student of Herbert Simon), Bruce 
Buchanan (a philosopher turned computer scientist), and Joshua Lederberg (a Nobel laureate 
geneticist) teamed up to solve the problem of inferring molecular structure from the information 
provided by a mass spectrometer. The input to the program consists of the elementary formula of 
the molecule (e.g., C^H^NCE), and the mass spectrum givingthe masses ofthe various fragments 
of the molecule generated when it is bombarded by an electron beam. For example, the mass 
spectrum might contain a peak at m- 15 corresponding to the mass of a methyl (CH3) fragment. 

The naive version of the program generated all possible structures consistent with the 
formula, and then predicted what mass spectrum would be observed for each, comparing this 
with the actual spectrum. As one might expect, this rapidly became intractable for decent-sized 
molecules. The DENDRAL researchers consulted analytical chemists and found that they worked 
by looking for well-known patterns of peaks in the spectrum that suggested common substructures 
in the molecule. For example, the following rule is used to recognize a ketone (C=0) subgroup: 
if there are two peaks at a'i and x-> such that 

(a) x\ + x 2 = M + 28 (M is the mass ofthe whole molecule); 

(b) A'i — 28 is a high peak; 

(c) X2 — 28 is a high peak; 

(d) At least one of.vi and.v> is high, 
then there is a ketone subgroup 

Having recognized that the molecule contains a particular substructure, the number of possible 
candidates is enormously reduced. The DENDRAL team concluded that the new system was 
powerful because 

All the relevant theoretical knowledge to solve these problems has been mapped over from its 
general form in the [spectrum prediction component] ("first principles") to efficient special 
forms ("cookbook recipes"). (Feigenbaum etal., 1971) 

The significance of DENDRAL was that it was arguably the first successful knowledge-intensive 
system: its expertise derived from large numbers of special-purpose rules. Later systems also 
incorporated the main theme of McCarthy's Advice Taker approach — the clean separation of the 
knowledge (in the form of rules) and the reasoning component. 

With this lesson in mind, Feigenbaum and others at Stanford began the Heuristic Program- 
expert systems ming Project (HPP), to investigate the extent to which the new methodology of expert systems 
could be applied to other areas of human expertise. The next major effort was in the area of 
medical diagnosis. Feigenbaum, Buchanan, and Dr. Edward Shortliffe developed MYCIN to 
diagnose blood infections. With about 450 rules, MYCIN was able to perform as well as some 
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experts, and considerably better thanjunior doctors. It also contained two major differences from 
DENDRAL. First, unlike the DENDRAL rules, no general theoretical model existed from which the 
MYC1N rules could be deduced. They had to be acquired from extensive interviewing of experts, 
who in turn acquired them from direct experience of cases. Second, the rules had to reflect the 
uncertainty associated with medical knowledge. MYCIN incorporated a calculus of uncertainty 
called certainty factors (see Chapter 14), which seemed (at the time) to fit well with how doctors 
assessed the impact of evidence on the diagnosis. 

Other approaches to medical diagnosis were also followed. At Rutgers University, Saul 
Amarel’s Computers in Biomedicine project began an ambitious attempt to diagnose diseases 
based on explicit knowledge of the causal mechanisms of the disease process. Meanwhile, large 
groups at MIT and the New England Medical Center were pursuing an approach to diagnosis and 
treatment based on the theories of probability and utility. Their aim was to build systems that 
gave provably optimal medical recommendations. In medicine, the Stanford approach using rules 
provided by doctors proved more popular at first. But another probabilistic reasoning system, 
PROSPECTOR (Duda et al . , 1979), generated enormous publicity by recommending exploratory 
drilling at a geological site that proved to contain a large molybdenum deposit. 

The importance of domain knowledge was also apparent in the area of understanding 
natural language. Although Winograd's SHRDLU system for understanding natural language had 
engendered a good deal of excitement, its dependence on syntactic analysis caused some of 
the same problems as occurred in the early machine translation work. It was able to overcome 
ambiguity and understand pronoun references, but this was mainly because it was designed 
specifically for one area — the blocks world. Several researchers, including Eugene Chamiak, 
a fellow graduate student of Winograd's at MIT, suggested that robust language understanding 
would require general knowledge about the world and a general method for using that knowledge. 

At Yale, the linguist-turned- Al-researcher Roger Schank emphasized this point by claiming, 
"There is no such thing as syntax," which upset a lot of linguists, but did serve to start a useful 
discussion. Schank and his students built a series of programs (Schank and Abelson, 1977; 
Schank and Riesbeck, 1981; Dyer, 1983) that all had the task of understanding natural language. 
The emphasis, however, was less on language per se and more on the problems of representing 
and reasoning with the knowledge required for language understanding. The problems included 
representing stereotypical situations (Cullingford, 1981), describing human memory organization 
(Rieger, 1976; Kolodner, 1983), and understanding plans and goals (Wilensky, 1983). William 
Woods (1973) built the LUNAR system, which allowed geologists to ask questions in English 
about the rock samples brought back by the Apollo moon mission. LUNAR was the first natural 
language program that was used by people other than the system's author to get real work done. 
Since then, many natural language programs have been used as interfaces to databases. 

The widespread growth of applications to real-world problems caused a concomitant in- 
crease in the demands for workable knowledge representation schemes. A large number of 
different representation languages were developed. Some were based on logic — for example, 
the Prolog language became popular in Europe, and the PLANNER family in the United States. 
Others, following Minsky's idea of frames (1975), adopted a rather more structured approach, 
collecting together facts about particular object and event types, and arranging the types into a 
large taxonomic hierarchy analogous to a biological taxonomy. 
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AI becomes an industry (1980-1988) 

The first successful commercial expert system, Rl, began operation at Digital Equipment Cor- 
poration (McDermott, 1982). The program helped configure orders for new computer systems, 
and by 1986, it was saving the company an estimated $40 million a year. By 1988, DEC's AI 
group had 40 deployed expert systems, with more on the way. Du Pont had 100 in use and 500 in 
development, saving an estimated $10 million a year. Nearly every major U.S. corporation had 
its own AI group and was either using or investigating expert system technology. 

In 1981, the Japanese announced the "Fifth Generation" project, a 10-year plan to build 
intelligent computers running Prolog in much the same way that ordinary computers run machine 
code. The idea was that with the ability to make millions of inferences per second, computers 
would be able to take advantage of vast stores of rules. The project proposed to achieve full-scale 
natural language understanding, among other ambitious goals. 

The Fifth Generation project fueled interest in AI, and by taking advantage of fears of 
Japanese domination, researchers and corporations were able to generate support for a similar 
investment in the United States. The Microelectronics and Computer Technology Corporation i 
(MCC) was formed as a research consortium to counter the Japanese project. In Britain, the 
Alvey report reinstated the funding that was cut by the Lighthill report. 16 In both cases, AI was 
part of a broad effort, including chip design and human-interface research. 

The booming AI industry also included companies such as Carnegie Group, Inference, 
Intellicorp, and Teknowledge that offered the software tools to build expert systems, and hard- 
ware companies such as Lisp Machines Inc., Texas Instruments, Symbolics, and Xerox that 
were building workstations optimized for the development of Lisp programs. Over a hundred 
companies built industrial robotic vision systems. Overall, the industry went from a few million i 
in sales in 1980 to $2 billion in 1988. 



The return of neural networks (1986-present) 

Although computer science had neglected the field of neural networks after Minsky and Papert's 
Perceptrons book, work had continued in other fields, particularly physics. Large collections 
of simple neurons could be understood in much the same way as large collections of atoms in 
solids. Physicists such as Hopfield (1982) used techniques from statistical mechanics to analyze 
the storage and optimization properties of networks, leading to significant cross-fertilization of j 
ideas. Psychologists including David Rumelhart and Geoff Hinton continued the study of neural 
net models of memory. As we discuss in Chapter 19, the real impetus came in the mid-1980s 
when at least four different groups reinvented the back-propagation learning algorithm first found 
in 1 969 by Bryson and Ho. The algorithm was applied to many learning problems in computer 
science and psychology, and the widespread dissemination of the results in the collection Parallel 
Distributed Processing (Rumelhart and McClelland, 1986) caused great excitement. 

At about the same time, some disillusionment was occurring concerning the applicability 
of the expert system technology derived from MYCiN-type systems.- Many corporations and 



16 To save embarrassment, a new field called IKBS (Intelligent Knowledge-BasedSystems) was defined because Artificial 
Intelligence had been officially cancelled. 
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research groups found that building a successful expert system involved much more than simply 
buying a reasoning system and filling it with rules. Some predicted an “AI Winter" in which AI 
funding would be squeezed severely. It was perhaps this fear, and the historical factors on the 
neural network side, that led to a period in which neural networks and traditional AI were seen 
as rival fields, rather than as mutually supporting approaches to the same problem. 



Recent events (1987-present) 

Recent years have seen a sea change in both the content and the methodology of research in 
artificial intelligence. 17 It is now more common to build on existing theories than to propose 
brand new ones, to base claims on rigorous theorems or hard experimental evidence rather than 
on intuition, and to show relevance to real-world applications rather than toy examples. 

The field of speech recognition illustrates the pattern. In the 1970s, a wide variety of 
different architectures and approaches were tried. Many of these were rather ad hoc and fragile, 
and were demonstrated on a few specially selected examples. In recent years, approaches based 
on hidden Markov models (HMMs) have come to dominate the area. Two aspects of HMMs are 
relevant to the present discussion. First, they are based on a rigorous mathematical theory. This 
has allowed speech researchers to build on several decades of mathematical results developed in 
other fields. Second, they are generated by a process of training on a large corpus of real speech 
data. This ensures that the performance is robust, and in rigorous blind tests the HMMs have 
been steadily improving their scores. Speech technology and the related field of handwritten 
character recognition are already making the transition to widespread industrial and consumer 
applications. 

Another area that seems to have benefitted from formalization is planning. Early work by 
Austin Tate (1977), followed up by David Chapman (1987), has resulted in an elegant synthesis 
of existing planning programs into a simple framework. There have been a number of advances 
that built upon each other rather than starting from scratch each time. The result is that planning 
systems that were only good for microworlds in the 1970s are now used for scheduling of factory 
work and space missions, among other things. See Chapters 1 1 and 12 for more details. 

Judea Pearl's (1988) Probabilistic Reasoning in Intelligent Systems marked a new accep- 
tance of probability and decision theory in AI, following a resurgence of interest epitomized by 
Peter Cheeseman's ( 1 985 ) article "In Defense ofProbability." The belief network formalism was 
invented to allow efficient reasoning about the combination of uncertain evidence. This approach 
largely overcomes the problems with probabilistic reasoning systems of the 1960s and 1970s, 
and has come to dominate AI research on uncertain reasoning and expert systems. Work by 
Judea Pearl ( 1982a) and by Eric Horvitz and David Heckerman (Horvitz and Heckerman, 1986; 
Horvitz et al„ 1986) promoted the idea of normative expert systems: ones that act rationally 
according to the laws of decision theory and do not try to imitate human experts. Chapters 14 to 
16 cover this area. 



17 Some have characterized this change as a victory of the neats — those who think that AI theories should be grounded 
in mathematical rigor — over the scruflies — those who would rather try out lots of ideas, write some programs, and then 
assess what seems to be working. Both approaches are important. A shift toward increased neatness implies that the field 
has reached a level of stability and maturity. (Whether that stability will be disrupted by a new scruffy idea is another 
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Similar gentle revolutions have occurred in robotics, computer vision, machine learning 
(including neural networks), and knowledge representation. A better understanding of the prob- 
lems and their complexity properties, combined with increased mathematical sophistication, has 
led to workable research agendas and robust methods. Perhaps encouraged by the progress in 
solving the subproblems of AI, researchers have also started to look at the "whole agent" problem 
again. The work of Alien Newell, John Laird, and Paul Rosenbloom on SOAR (Newell, 1990; 
Laird et al . , 1 987) is the best-known example of a complete agent architecture in AI. The so-called 
"situated" movement aims to understand the workings of agents embedded in real environments 
with continuous sensory inputs. Many interesting results are coming out of such work, including 
the realization that the previously isolated subfields of AI may need to be reorganized somewhat 
when their results are to be tied together into a single agent design. 



1.4 THE STATE OF THE ART 



International grandmaster Arnold Denker studies the pieces on the board in front of him. He 
realizes there is no hope; he must resign the game. His opponent, HITECH, becomes the first 
computer program to defeat a grandmaster in a game of chess (Berliner, 1989). 

"I want to go from Boston to San Francisco,” the traveller says into the microphone. "What 
date will you be travelling on?" is the reply. The traveller explains she wants to go October 20th, 
nonstop, on the cheapest available fare, returning on Sunday. A speech understanding program 
named PEGASUS handles the whole transaction, which results in a confirmed reservation that 
saves the traveller $894 over the regular coach fare. Even though the speech recognizer gets one 
out of ten words wrong, 18 it is able to recover from these errors because of its understanding of 
how dialogs are put together (Zue et al ., 1994). 

An analyst in the Mission Operations room of the Jet Propulsion Laboratory suddenly 
starts paying attention. A red message has flashed onto the screen indicating an "anomaly" with 
the Voyager spacecraft, which is somewhere in the vicinity of Neptune. Fortunately, the analyst 
is able to correct the problem from the ground. Operations personnel believe the problem might 
have been overlooked had it not been for MARVEL, a real-time expert system that monitors the 
massive stream of data transmitted by the spacecraft, handling routine tasks and alerting the 
analysts to more serious problems (Schwuttke, 1992). 

Cruising the highway outside of Pittsburgh at a comfortable 55 mph, the man in the driver's 
seat seems relaxed. He should be — forthe past 90 miles, he has not had to touch the steering wheel, 
brake, or accelerator. The real driver is a robotic system that gathers input from video cameras, 
sonar, and laser range finders attached to the van. It combines these inputs with experience 
learned from training runs and succesfully computes how to steer the vehicle (Pomerleau, 1993). 

A leading expert on lymph-node pathology describes a fiendishly difficult case to the 
expert system, and examines the system's diagnosis. He scoffs at the system's response. Only 
slightly worried, the creators of the system suggest he ask the computer for an explanation of 



18 Some other existing systems err only half as often on this task. 
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the diagnosis. The machine points out the major factors influencing its decision, and explains 
the subtle interaction of several of the symptoms in this case. The expert admits his error, 
eventually (Heckerman, 1991). 

From a camera perched on a street light above the crossroads, the traffic monitor watches 
the scene. If any humans were awake to read the main screen, they would see "Citroen 2CV 
turning from Place de la Concorde into Champs Ely sees," "Large truck of unknown make stopped 
on Place de la Concorde," and so on into the night. And occasionally, "Major incident on Place 
de la Concorde, speeding van collided with motorcyclist," and an automatic call to the emergency 
services (King et al„ 1993; Roller etal., 1994). 

These are just a few examples of artificial intelligence systems that exist today. Not magic 
or science fiction — butrather science, engineering, and mathematics, to which this book provides 
an introduction. 



1L5 SUMMARY 



This chapter defines AI and establishes the cultural background against which it has developed. 

Some of the important points are as follows: 

• Different people think of AI differently. Two important questions to ask are: Are you 
concerned with thinking or behavior? Do you want to model humans, or work from an 
ideal standard? 

• In this book, we adopt the view that intelligence is concerned mainly with rational action. 
Ideally, an intelligent agent takes the best possible action in a situation. We will study the 
problem of building agents that are intelligent in this sense. 

• Philosophers (going back to 400 B.C.) made AI conceivable by considering the ideas that 
the mind is in some ways like a machine, that it operates on knowledge encoded in some 
internal language, and that thought can be used to help arrive at the right actions to take. 

• Mathematicians provided the tools to manipulate statements of logical certainty as well 
as uncertain, probabilistic statements. They also set the groundwork for reasoning about 
algorithms. 

• Psychologists strengthened the idea that humans and other animals can be considered 
information processing machines. Linguists showed that language use fits into this model. 

• Computer engineering provided the artifact that makes AI applications possible. AI pro- 
grams tend to be large, and they could not work without the great advances in speed and 
memory that the computer industry has provided. 

• The history of AI has had cycles of success, misplaced optimism, and resulting cutbacks 
in enthusiasm and funding. There have also been cycles of introducing new creative 
approaches and systematically refining the best ones. 

• Recent progress in understanding the theoretical basis for intelligence has gone hand in 
hand with improvements in the capabilities of real systems. 
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BIBLIOGRAPHICAL AND HISTORICAL NOTES 

Daniel Crevier's (1993 ) Artificial Intelligence gives a complete history of the field, and Raymond 
Kurzweil’s (1990) Age of Intelligent Machines situates AI in the broader context of computer 
science and intellectual history in general. Dianne Martin (1993) documents the degree to which 
early computers were endowed by the media with mythical powers of intelligence. 

The methodological status of artificial intelligence is discussed in The Sciences of the Ar- 
tificial, by Herb Simon (1981 ), which discusses research areas concerned with complex artifacts. 
It explains how AI can be viewed as both science and mathematics. 

Artificial Intelligence: The Very Idea, by John Haugeland (1985) gives a readable account of 
the philosophical and practical problems of AI. Cognitive science is well-described by Johnson- 
Laird's The Computer and the Mind: An Introduction to Cognitive Science. Baker (1989) 
covers the syntactic part of modern linguistics, and Chierchia and McConnell-Ginet ( 1 990) cover 
semantics. Alien (1995) covers linguistics from the AI point of view. 

Early AI work is covered in Feigenbaum and Feldman's Computers and Thought, Minsky's 
Semantic Information Processing, and the Machine Intelligence series edited by Donald Michie. 
A large number of influential papers are collected in Readings in Artificial Intelligence (Webber 
and Nilsson, 1981 ). Early papers on neural networks are collected in Neurocomputing (Anderson 
and Rosenfeld, 1 988). The Encyclopedia o/ A/Shapiro, 1992) contains survey articles on almost 
every topic in AI. These articles usually provide a good entry point into the research literature on 
each topic. The four-volume Handbook of Artificial Intelligence (Barr and Feigenbaum, 1981) 
contains descriptions of almost every major AI system published before 1981. 

The most recent work appears in the proceedings of the major AI conferences: the biennial 
International Joint Conference on AI (IJCAI), and the annual National Conference on AI, more 
often known as AAAI, after its sponsoring organization. The major journals for general AI are 
Artificial Intelligence, Computational Intelligence, the IEEE Transactions on Pattern Analysis 
and Machine Intelligence, and the electronic Journal of Artificial Intelligence Research. There 
are also many journals devoted to specific areas, which we cover in the appropriate chapters. 
Commercial products are covered in the magazines AI Expert and PCAI. The main professional 
societies for AI are the American Association for Artificial Intelligence (AAAI), the ACM Special 
Interest Group in Artificial Intelligence (SIGART), and the Society for Artificial Intelligence and 
Simulation of Behaviour (AISB). AAAI’s AI Magazine and the SIGART Bulletin contain many 
topical and tutorial articles as well as announcements of conferences and workshops. 



EXERCISES 



i. 




These exercises are intended to stimulate discussion, and some might be set as term projects. 
Alternatively, preliminary attempts can be made now, and these attempts can be reviewed after 
completing the book. 

1.1 Read Turing's original paper on AI (Turing, 1950). In the paper, he discusses several 
potential objections to his proposed enterprise and his test for intelligence. Which objections 
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still carry some weight? Are his refutations valid? Can you think of new objections arising from 
developments since he wrote the paper? In the paper, he predicts that by the year 2000, a computer 
will have a 30% chance of passing a five-minute Turing Test with an unskilled interrogator. Do 
you think this is reasonable? 

12 We characterized the definitions of AI along two dimensions, human vs. ideal and thought 
vs. action. But there are other dimensions that are worth considering. One dimension is whether 
we are interested in theoretical results or in practical applications. Another is whether we intend 
our intelligent computers to be conscious or not. Philosophers have had a lot to say about this 
issue, and although most AI researchers are happy to leave the questions to the philosophers, 
STRONG AI there has been heated debate. The claim that machines can be conscious is called the strong AI 

weak ai claim; the weak AI position makes no such claim. Characterize the eight definitions on page 

5 and the seven following definitions according to the four dimensions we have mentioned and 
whatever other ones you feel are helpful. 

Artificial intelligence is . . . 

a. "a collection of algorithms that are computationally tractable, adequate approximations of 
intractably specified problems" (Partridge, 1991) 

b. "the enterprise of constructing a physical symbol system that can reliably pass the Turing 
Test" (Ginsberg, 1993) 

c. "the field of computer science that studies how machines can be made to act intelli- 
gently" (Jackson, 1986) 

d. "a field of study that encompasses computational techniques for performing tasks that 
apparently require intelligence when performed by humans" (Tanimoto, 1990) 

e. "a very general investigation of the nature of intelligence and the principles and mechanisms 
required for understanding or replicating it" (Sharpies et al., 1989) 

f. "the getting of computers to do things that seem to be intelligent" (Rowe, 1988). 

1.3 There are well-known classes of problems that are intractably difficult for computers, 
and other classes that are provably undecidable by any computer. Does this mean that AI is 
impossible? 

1.4 Suppose we extend Evans's ANALOGY program so that it can score 200 on a standard IQ 
test. Would we then have a program more intelligent than a human? Explain. 

1.5 Examine the AI literature to discover whether or not the following tasks can currently be 
solved by computers: 

a. Playing a decent game of table tennis (ping-pong). 

b. Driving in the center of Cairo. 

c. Playing a decent game of bridge at a competitive level. 

d. Discovering and proving new mathematical theorems. 

e. Writing an intentionally funny story. 

f. Giving competent legal advice in a specialized area of law. 

g. Translating spoken English into spoken Swedish in real time. 
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For the currently infeasible tasks, try to find out what the difficulties are and estimate when they 
will be overcome. 

1.6 Find an article written by a lay person in a reputable newspaper or magazine claiming 
the achievement of some intelligent capacity by a machine, where the claim is either wildly 
exaggerated or false. 

1.7 Fact, fiction, and forecast: 

a. Find a claim in print by a reputable philosopher or scientist to the effect that a certain 
capacity will never be exhibited by computers, where that capacity has now been exhibited. 

b. Find a claim by a reputable computer scientist to the effect that a certain capacity would 
be exhibited by a date that has since passed, without the appearance of that capacity. 

c. Compare the accuracy of these predictions to predictions in other fields such as biomedicine, 
fusion power, nanotechnology, transportation, or home electronics. 

1.8 Some authors have claimed that perception and motor skills are the most important part of 
intelligence, and that "higher-level" capacities are necessarily parasitic — simple add-ons to these 
underlying facilities. Certainly, most of evolution and a large part of the brain have been devoted 
to perception and motor skills, whereas AI has found tasks such as game playing and logical 
inference to be easier, in many ways, than perceiving and acting in the real world. Do you think 
that AI’s traditional focus on higher-level cognitive abilities is misplaced? 

1.9 "Surely computers cannot be intelligent — they can only do what their programmers tell 
them." Is the latter statement true, and does it imply the former? 

1.10 "Surely animals cannot be intelligent — they can only do what their genes tell them." Is 
the latter statement true, and does it imply the former? 




INTELLIGENT AGENTS 



In which we discuss what an intelligent agent does, how it is related to its environment, 
how it is evaluated, and how we might go about building one. 



2.1 INTRODUCTION 



An agent is anything that can be viewed as perceiving its environment through sensors and acting 
upon that environment through effectors. A human agent has eyes, ears, and other organs for 
sensors, and hands, legs, mouth, and other body parts for effectors. A robotic agent substitutes 
cameras and infrared range finders for the sensors and various motors for the effectors. A 
software agent has encoded bit strings as its percepts and actions. A generic agent is diagrammed 
in Figure 2.1. 

Our aim in this book is to design agents that do a good job of acting on their environment. 
First, we will be a little more precise about what we mean by a good job. Then we will talk about 
different designs for successful agents — filling in the question mark in Figure 2.1. We discuss 
some of the general principles used in the design of agents throughout the book, chief among 
which is the principle that agents should know things. Finally, we show how to couple an agent 
to an environment and describe several kinds of environments. 



12 How AGFNTS SHOUI.D ACT 



■ ha tional AGENT A rational agent is one that does the right thing. Obviously, this is better than doing the wrong 

■ thing, but what does it mean? As a first approximation, we will say that the right action is the 

I one that will cause the agent to be most successful. That leaves us with the problem of deciding 

■ how and when to evaluate the agent's success. 

3i 
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m!asure ANCE Wc use the term performance measure for the how — the criteria that determine how 

successful an agent is. Obviously, there is not one fixed measure suitable for all agents. We 
could ask the agent for a subjective opinion of how happy it is with its own performance, but 
some agents would be unable to answer, and others would delude themselves. (Human agents in 
particular are notorious for "sour grapes” — saying they did not really want something after they 
are unsuccessful at getting it.) Therefore, we will insist on an objective performance measure 
imposed by some authority. In other words, we as outside observers establish a standard of what 
it means to be successful in an environment and use it to measure the performance of agents. 

As an example, consider the case of an agent that is supposed to vacuum a dirty floor. A 
plausible performance measure would be the amount of dirt cleaned up in a single eight-hour shift. 
A more sophisticated performance measure would factor in the amount of electricity consumed 
and the amount of noise generated as well. A third performance measure might give highest 
marks to an agent that not only cleans the floor quietly and efficiently, but also finds time to go 
windsurfing at the weekend. 1 

The when of evaluating performance is also important. If we measured how much dirt the 
agent had cleaned up in the first hour of the day, we would be rewarding those agents that start 
fast (even if they do little or no work later on), and punishing those that work consistently. Thus, 
we want to measure performance over the long run, be it an eight-hour shift or a lifetime. 
omniscience We need to be careful to distinguish between rationality and omniscience. An omniscient 

agent knows the actual outcome of its actions, and can act accordingly; but omniscience is : 
impossible in reality. Consider the following example: I am walking along the Champs Elysees 
one day and I see an old friend across the street. There is no traffic nearby and I'm not otherwise 
engaged, so, being rational, I start to cross the street. Meanwhile, at 33,000 feet, a cargo door 
falls off a passing airliner, 2 and before I make it to the other side of the street I am flattened. Was 
I irrational to cross the street? It is unlikely that my obituary would read "Idiot attempts to cross 



1 There is a danger here for those who establish performance measures: you often get what you ask for. That is. if 
you measure success by the amount of dirt cleaned up, then some clever agent is bound to bring in a load of dirt each 
morning, quickly clean it up, and get a good performance score. What you really want to measure is how clean the floor 
is, but determining that is more difficult than just weighing the dirt cleaned up. 

2 See N. Henderson. "New door latches urged for Boeing 747 jumbo jets." Washington Post, 8/24/89. 
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• The performance measure that defines degree of success. 

• Everything that the agent has perceived so far. We will call this complete perceptual history 

the percept sequence. 

• What the agent knows about the environment. 

• The actions that the agent can perform. 

This leads to a definition of an ideal rational agent: For each possible percept sequence, an 
ideal rational agent should do whatever action is expected to maximize its performance measure, 
on the basis of the evidence provided by the percept sequence and whatever built-in knowledge 
the agent has. 

We need to look carefully at this definition. At first glance, it might appear to allow an 
agent to indulge in some decidedly underintelligent activities. For example, if an agent does not 
look both ways before crossing a busy road, then its percept sequence will not tell it that there is 
a large truck approaching at high speed. The definition seems to say that it would be OK for it to 
cross the road. In fact, this interpretation is wrong on two counts. First, it would not be rational 
to cross the road: the risk of crossing without looking is too great. Second, an ideal rational 
agent would have chosen the "looking" action before stepping into the street, because looking 
helps maximize the expected performance. Doing actions in order to obtain useful information 
is an important part of rationality and is covered in depth in Chapter 16. 

The notion of an agent is meant to be a tool for analyzing systems, not an absolute 
characterization that divides the world into agents and non-agents. Consider a clock. It can be 
thought of as just an inanimate object, or it can be thought of as a simple agent. As an agent, 
most clocks always do the right action: moving their hands (or displaying digits) in the proper 
fashion. Clocks are a kind of degenerate agent in that their percept sequence is empty; no matter 
what happens outside, the clock's action should be unaffected. 

Well, this is not quite true. If the clock and its owner take a trip from California to Australia, 
the right thing for the clock to do would be to turn itself back six hours. We do not get upset at 
our clocks for failing to do this because we realize that they are acting rationally, given their lack 
of perceptual equipment. 3 
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MAPPING 



I 



The ideal mapping from percept sequences to actions 

Once we realize that an agent's behavior depends only on its percept sequence to date, then we can 
describe any particular agent by making a table of the action it takes in response to each possible 
percept sequence. (For most agents, this would be a very long list — infinite, in fact, unless we 
place a bound on the length of percept sequences we want to consider.) Such a list is called 
a mapping from percept sequences to actions. We can, in principle, find out which mapping 
correctly describes an agent by trying out all possible percept sequences and recording which 
actions the agent does in response. (If the agent uses some randomization in its computations, 
then we would have to try some percept sequences several times to get a good idea of the agent's 
average behavior.) And if mappings describe agents, then ideal mappings describe ideal agents. 
Specifying which action an agent ought to take in response to any given percept sequence provides 
a design for an ideal agent. 

This does not mean, of course, that we have to create an explicit table with an entry 
for every possible percept sequence. It is possible to define a specification of the mapping 
without exhaustively enumerating it. Consider a very simple agent: the square-root function 
on a calculator. The percept sequence for this agent is a sequence of keystrokes representing a 
number, and the action is to display a number on the display screen. The ideal mapping is that 
when the percept is a positive number x, the right action is to display a positive number z such 
that 7} % x, accurate to, say, 15 decimal places. This specification of the ideal mapping does 
not require the designer to actually construct a table of square roots. Nor does the square-root 
function have to use a table to behave correctly: Figure 2.2 shows part of the ideal mapping and 
a simple program that implements the mapping using Newton's method. 

The square-root example illustrates the relationship between the ideal mapping and an 
ideal agent design, for a very restricted task. Whereas the table is very large, the agent is a nice, 
compact program. It turns out that it is possible to design nice, compact agents that implement 



Percept x 


Action z 


1.0 


1.000000000000000 


1.1 


1.048808848170152 


1.2 


1.095445115010332 


1.3 


1.140175425099138 


1.4 


1.183215956619923 


1.5 


1.224744871391589 


1.6 


1.264911064067352 


1.7 


1.303840481040530 


1.8 


1.341640786499874 


1.9 


1.378404875209022 

i ; 



function SQRT(x) 

z <— 1 .0 / * initial guess * / 

repeat until \z 2 - x] < 10“ 15 
z <- z - (z 2 - x)/(2z) 
end 

return z 



Figure 2.2 Part of the ideal mapping for the square-root problem (accurate to 1 5 digits), and ; 
corresponding program that implements the ideal mapping. 




Section 2.3- Structure of Intelligent Agents 35 



the ideal mapping for much more general situations: agents that can solve a limitless variety of 
tasks in a limitless variety of environments. Before we discuss how to do this, we need to look 
at one more requirement that an intelligent agent ought to satisfy. 



Autonomy 

There is one more thing to deal with in the definition of an ideal rational agent: the "built-in 
knowledge" part. If the agent's actions are based completely on built-in knowledge, such that it 
AUTONOMY need pay no attention to its percepts, then we say that the agent lacks autonomy. For example, 

if the clock manufacturer was prescient enough to know that the clock’s owner would be going 
to Australia at some particular date, then a mechanism could be built in to adjust the hands 
automatically by six hours atjust the right time. This would certainly be successful behavior, but 
the intelligence seems to belong to the clock's designer rather than to the clock itself. 

An agent's behavior can be based on both its own experience and the built-in knowledge 
, used in constructing the agent for the particular environment in which it operates. A system is 

I - autonomous 4 to the extent that its behavior is determined bx its own experience. It would be 

too stringent, though, to require complete autonomy from the word go: when the agent has had 
little or no experience, it would have to act randomly unless the designer gave some assistance. 
So, just as evolution provides animals with enough built-in reflexes so that they can survive long 
enough to learn for themselves, it would be reasonable to provide an artificial intelligent agent 
with some initial knowledge as well as an ability to learn. 

Autonomy not only fits in with our intuition, but it is an example of sound engineering 
practices. An agent that operates on the basis of built-in assumptions will only operate success- 
fully when those assumptions hold, and thus lacks flexibility. Consider, for example, the lowly 
dung beetle. After digging its nest and laying its eggs, it fetches a ball of dung from a nearby heap 
to plug the entrance; if the ball of dung is removed from its grasp en route, the beetle continues 
on and pantomimes plugging the nest with the nonexistent dung ball, never noticing that it is 
missing. Evolution has built an assumption into the beetle's behavior, and when it is violated, 
unsuccessful behavior results. A truly autonomous intelligent agent should be able to operate 
successfully in a wide variety of environments, given sufficient time to adapt. 



2.3 STRUCTURE OF INTF.I.I.IGF.NT AGF.NTS 



So far we have talked about agents by describing their behavior — the action that is performed 
after any given sequence of percepts. Now, we will have to bite the bullet and talk about how 
agent program the insides work. The job of AI is to design the agent program: a function that implements 
the agent mapping from percepts to actions. We assume this program will run on some sort of 
architecture computing device, which we will call the architecture. Obviously, the program we choose has 




36 



Chapter 2. Intelligent Agents 



to be one that the architecture will accept and run. The architecture might be a plain computer, or 
it might include special-purpose hardware for certain tasks, such as processing camera images or 
filtering audio input. It might also include software that provides a degree of insulation between 
the raw computer and the agent program, so that we can program at a higher level. In general, 
the architecture makes the percepts from the sensors available to the program, runs the program, 
and feeds the program's action choices to the effectors as they are generated. The relationship 
among agents, architectures, and programs can be summed up as follows: 

agent = architecture + program 

Most of this book is about designing agent programs, although Chapters 24 and 25 deal directly 
with the architecture. 

Before we design an agent program, we must have a pretty good idea of the possible 
percepts and actions, what goals or performance measure the agent is supposed to achieve, and 
what sort of environment it will operate in. 5 These come in a wide variety. Figure 2.3 shows the 
basic elements for a selection of agent types. 

It may come as a surprise to some readers that we include in our list of agent types some 
programs that seem to operate in the entirely artificial environment defined by keyboard input 
and character output on a screen. "Surely," one might say, "this is not a real environment, is 
it?" In fact, what matters is not the distinction between "real" and "artificial" environments, 
but the complexity of the relationship among the behavior of the agent, the percept sequence 
generated by the environment, and the goals that the agent is supposed to achieve. Some "real" 
environments are actually quite simple. For example, a robot designed to inspect parts as they 
come by on a conveyer belt can make use of a number of simplifying assumptions: that the 
lighting is always just so, that the only thing on the conveyer belt will be parts of a certain kind, 
and that there are only two actions — accept the part or mark it as a reject. 
software agents In contrast, some software agents (or software robots or softbots) exist in rich, unlimited 

softbots domains. Imagine a softbot designed to fly a flight simulator for a 747. The simulator is a 

very detailed, complex environment, and the software agent must choose from a wide variety of 
actions in real time. Or imagine a softbot designed to scan online news sources and show the 
interesting items to its customers. To do well, it will need some natural language processing 
abilities, it will need to learn what each customer is interested in, and it will need to dynamically 
change its plans when, for example, the connection for one news source crashes or a new one 
comes online. 

Some environments blur the distinction between "real" and "artificial." In the ALIVE 
environment (Maes etal., 1994), software agents are given as percepts a digitized camera image 
of a room where a human walks about. The agent processes the camera image and chooses an 
action. The environment also displays the camera image on a large display screen that the human 
can watch, and superimposes on the image a computer graphics rendering of the software agent. 
One such image is a cartoon dog, which has been programmed to move toward the human (unless 
he points to send the dog away) and to shake hands or jump up eagerly when the human makes 
certain gestures. 

5 For the acronymically minded, we call this the PAGE (Percepts, Actions, Goals, Environment) description. Note that 
the goals do not necessarily have to be represented within the agent; they simply describe the performance measure by 
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Agent Type 


Percepts 


Actions 


Goals 


Environment 


Medical diagnosis 
system 


Symptoms, 
findings, patient's 
answers 


Questions, tests, 
treatments 


Healthy patient, 
minimize costs 


Patient, hospital 


Satellite image 
analysis system 


Pixels of varying 
intensity, color 


Print a 

categorization of 


categorization 


Images from 
orbiting satellite 


Part-picking robot 


Pixels of varying 
intensity 


Pick up parts and 


Place parts in j 

correct bins 


Conveyor belt 
with parts 


Refinery controller 


Temperature, 
pressure readings 


Open, close 
valves; adjust 
temperature 


Maximize purity, 
yield, safety 

1 


Refinery 


Interactive English 


Typed words 


Print exercises, 

suggestions, 

corrections 


Maximize 
student's score on 


Set of students 



Figure 2.3 Examples of agent types and their PAGE descriptions. 



The most famous artificial environment is the Turing Test environment, in which the whole 
point is that real and artificial agents are on equal footing, but the environment is challenging 
enough that it is very difficult for a software agent to do as well as a human. Section 2.4 describes 
in more detail the factors that make some environments more demanding than others. 



Agent programs 



We will be building intelligent agents throughout the book. They will all have the same skeleton, 
namely, accepting percepts from an environment and generating actions. The early versions of 
agent programs will have a very simple form (Figure 2.4). Each will use some internal data 
structures that will be updated as new percepts arrive. These data structures are operated on by 
the agent's decision-making procedures to generate an action choice, which is then passed to the 
architecture to be executed. 

There are two things to note about this skeleton program. First, even though we defined 
the agent mapping as a function from percept sequences to actions, the agent program receives 
only a single percept as its input. It is up to the agent to build up the percept sequence in memory, 
if it so desires. In some environments, it is possible to be quite successful without storing 
the percept sequence, and in complex domains, it is infeasible to store the complete sequence. 
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function Skeleton-Agent( perceprreturns action 
static: memory, the agent's memory of the world 

memory — Update-Memory (memory, percept) 
action — Choose-Best-Action(wc»i«/t) 
memory — UPDATE-MEMORY(/ne«!wy, action) 
return action 



Figure 2.4 A skeleton agent. On each invocation, the agent's memory is updated to reflect 
the new percept, the best action is chosen, and the fact that the action was taken is also stored in 
memory. The memory persists from one invocation to the next. 



Second, the goal or performance measure is not part of the skeleton program. This is because 
the performance measure is applied externally to judge the behavior of the agent, and it is often 
possible to achieve high performance without explicit knowledge of the performance measure 
(see, e.g., the square-root agent). 



Why not just look up the answers? 

Let us start with the simplest possible way we can think of to write the agent program — a lookup 
table. Figure 2.5 shows the agent program. It operates by keeping in memory its entire percept 
sequence, and using it to index into table, which contains the appropriate action for all possible 
percept sequences. 

It is instructive to consider why this proposal is doomed to failure: 

1 . The table needed for something as simple as an agent that can only play chess would be 
about 35 100 entries. 

2. It would take quite a long time for the designer to build the table. 

3. The agent has no autonomy at all. because the calculation of best actions is entirely built-in. ; 
So if the environment changed in some unexpected way, the agent would be lost. 



function Table-Driven-Agent( percept) returns action 
static: percepts, a sequence, initially empty 

table, a table, indexed by percept sequences, initially fully specified 

append percept to the end of percepts 
action <— Lookup! percept stable) 
return action 



Figure 2.5 An agent based on a prespecified lookup table. It keeps track of the percept 
sequence andjust looks up the best action. 
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4. Even if we gave the agent a learning mechanism as well, so that it could have a degree of 
autonomy, it would take forever to learn the right value for all the table entries. 

Despite all this, Table-Driven- Agent does do what we want: it implements the desired agent 
mapping. It is not enough to say, "It can't be intelligent;” the point is to understand why an agent 
that reasons (as opposed to looking things up in a table) can do even better by avoiding the four 
drawbacks listed here. 



An example 

At this point, it will be helpful to consider a particular environment, so that our discussion 
can become more concrete. Mainly because of its familiarity, and because it involves a broad 
range of skills, we will look at the job of designing an automated taxi driver. We should point 
out, before the reader becomes alarmed, that such a system is currently somewhat beyond the 
capabilities of existing technology, although most of the components are available in some form. 6 
The full driving task is extremely open-ended — there is no limit to the novel combinations of 
circumstances that can arise (which is another reason why we chose it as a focus for discussion). 

We must first think about the percepts, actions, goals and environment for the taxi. They 
are summarized in Figure 2.6 and discussed in turn. 



Agent Type 


Percepts 


Actions 


Goals 


Environment 


Taxi driver 


Cameras, 

speedometer, GPS, 
sonar, microphone 


Steer, accelerate, 
brake, talk to 
passenger 


Safe, fast, legal, 
comfortable trip, 
maximize profits 


Roads, other 
traffic, pedestrians, 
customers 


| Figure 2.6 The taxi driver agent type. 



I 



The taxi will need to know where it is, what else is on the road, and how fast it is going. 
This information can be obtained from the percepts provided by one or more controllable TV 
cameras, the speedometer, and odometer. To control the vehicle properly, especially on curves, it 
should have an accelerometer; it will also need to know the mechanical state of the vehicle, so it 
will need the usual array of engine and electrical system sensors. It might have instruments that 
are not available to the average human driver: a satellite global positioning system (GPS) to give 
it accurate position information with respect to an electronic map; or infrared or sonar sensors to 
detect distances to other cars and obstacles. Finally, it will need a microphone or keyboard for 
the passengers to tell it their destination. 

The actions available to a taxi driver will be more or less the same ones available to a human 
driver: control over the engine through the gas pedal and control over steering and braking. In 
addition, it will need output to a screen or voice synthesizer to talk back to the passengers, and 
perhaps some way to communicate with other vehicles. 

6 See page 26 for a description of an existing driving robot, or look at the conference proceedings on Intelligent Vehicle 
and Highway Systems (IVHS). 
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What performance measure would we like our automated driver to aspire to? Desirable 
qualities include getting to the correct destination; minimizing fuel consumption and wear and 
tear; minimizing the trip time and/or cost; minimizing violations of traffic laws and disturbances 
to other drivers; maximizing safety and passenger comfort; maximizing profits. Obviously, some 
of these goals conflict, so there will be trade-offs involved. 

Finally, were this a real project, we would need to decide what kind of driving environment 
the taxi will face. Should it operate on local roads, or also on freeways? Will it be in Southern 
California, where snow is seldom a problem, or in Alaska, where it seldom is not? Will it always 
be driving on the right, or might we want it to be flexible enough to drive on the left in case we 
want to operate taxis in Britain or Japan? Obviously, the more restricted the environment, the 
easier the design problem. 

Now we have to decide how to build a real program to implement the mapping from 
percepts to action. We will find that different aspects of driving suggest different types of agent 
program. We will consider four types of agent program: 

• Simple reflex agents 

• Agents that keep track of the world 

• Goal-based agents 

• Utility-based agents 



Simple reflex agents 

The option of constructing an explicit lookup table is out of the question. The visual input from 
a single camera comes in at the rate of 50 megabytes per second (25 frames per second, 1000 x 
1000 pixels with 8 bits of color and 8 bits of intensity information). So the lookup table for an 
hour would be 2 60 x 60 x 50m entries. 

However, we can summarize portions of the table by noting certain commonly occurring 
input/output associations. For example, if the car in front brakes, and its brake lights come on, 
then the driver should notice this and initiate braking. In other words, some processing is done on 
the visual input to establish the condition we call "The car in front is braking"; then this triggers 
some established connection in the agent program to the action "initiate braking". We call such 
^notion-action a connection a condition-action rule 7 written as 

if car-in-front-is-brakingthen initiate-braking 

Humans also have many such connections, some of which are learned responses (as for driving) 
and some of which are innate reflexes (such as blinking when something approaches the eye). 
In the course of the book, we will see several different ways in which such connections can be 
learned and implemented. 

Figure 2.7 gives the structure of a simple reflex agent in schematic form, showing how 
the condition-action rules allow the agent to make the connection from percept to action. (Do 
not worry if this seems trivial; it gets more interesting shortly.) We use rectangles to denote 

7 Also called situation-action rules, productions, or if-then rules. The last term is also used by some authors for 
logical implications, so we will avoid it altogether. 
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function Simple-Reflex- Agent( percept) returns action 
static: rules, a set of condition-action rules 

state — Interpret-Input( percept) 
rule — RULE-MATCH(stote, rules) 
action <- RuLE-AcnoN[r«/e] 
return action 



Figure 2.8 A simple reflex agent. It works by finding a rule whose condition matches the 
current situation (as defined by the percept) and then doing the action associated with that rule. 



the current internal state of the agent's decision process, and ovals to represent the background 
information used in the process. The agent program, which is also very simple, is shown in 
Figure 2.8. The INTERPRET-INPUT function generates an abstracted description of the current 
state from the percept, and the RULE-MATCH function returns the first rule in the set of rules that 
matches the given state description. Although such agents can be implemented very efficiently 
(see Chapter 10), their range of applicability is very narrow, as we shall see. 



Agents that keep track of the world 

The simple reflex agent described before will work only if the correct decision can be made 
on the basis of the current percept. If the car in front is a recent model, and has the centrally 
mounted brake light now required in the United States, then it will be possible to tell if it is 
braking from a single image. Unfortunately, older models have different configurations of tail 
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lights, brake lights, and turn-signal lights, and it is not always possible to tell if the car is braking. 
Thus, even for the simple braking rule, our driver will have to maintain some sort of internal 
INTERNAL STATE state in order to choose an action. Here, the internal state is not too extensive — itjust needs the 
previous frame from the camera to detect when two red lights at the edge of the vehicle go on or 
off simultaneously. 

Consider the following more obvious case: from time to time, the driver looks in the 
rear-view mirror to check on the locations of nearby vehicles. When the driver is not looking in 
the mirror, the vehicles in the next lane are invisible (i.e., the states in which they are present and 
absent are indistinguishable); but in order to decide on a lane-change maneuver, the driver needs 
to know whether or not they are there. 

The problem illustrated by this example arises because the sensors do not provide access to 
the complete state of the world. In such cases, the agent may need to maintain some internal state 
information in order to distinguish between world states that generate the same perceptual input 
but nonetheless are significantly different. Here, "significantly different" means that different 
actions are appropriate in the two states. 

Updating this internal state information as time goes by requires two kinds of knowledge to 
be encoded in the agent program. First, we need some information about how the world evolves 
independently of the agent — for example, that an overtaking car generally will be closer behind 
than it was a moment ago. Second, we need some information about' how the agent's own actions 
affect the world — for example, that when the agent changes lanes to the right, there is a gap (at 
least temporarily) in the lane it was in before, or that after driving for five minutes northbound 
on the freeway one is usually about five miles north of where one was five minutes ago. 

Figure 2.9 gives the structure of the reflex agent, showing how the current percept is 
combined with the old internal state to generate the updated description of the current state. The 
agent program is shown in Figure 2.10. The interesting part is the function UPDATE-STATE, which 
is responsible for creating the new internal state description. As well as interpreting the new 
percept in the light of existing knowledge about the state, it uses information about how the world 
evolves to keep track of the unseen parts of the world, and also must know about what the agent's 
actions do to the state of the world. Detailed examples appear in Chapters 7 and 17. 



Goal-based agents 

Knowing about the current state of the environment is not always enough to decide what to do. 
For example, at a road junction, the taxi can turn left, right, or go straight on. The right decision 
depends on where the taxi is trying to get to. In other words, as well as a current state descriptions 
goal the agent needs some sort of goal information, which describes situations that are desirable — 

for example, being at the passenger's destination. The agent program can combine this with! 
information about the results of possible actions (the same information as was used to update; 
internal state in the reflex agent) in order to choose actions that achieve the goal. Sometimes 
this will be simple, when goal satisfaction results immediately from a single action; sometimes, 
it will be more tricky, when the agent has to consider long sequences of twists and turns to find 
search a way to achieve the goal. Search (Chapters 3 to 5) and planning (Chapters 11 to 13) are the 

planning subfields of AI devoted to finding action sequences that do achieve the agent's goals. 
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function Reflex-Agent-With-State( percept) returns action 
static: state, a description of the current world state 
rules, a set of condition-action rules 

state ^Update-State (state, percept) 
rule — RULE-MATCH(state, rules ) 
action — RULE-ACTlON[rw/e] 
state <— U PDATE-STATE(.stafe, action ) 
return action 



Figure 2.10 A reflex agent with internal state. It works by finding a rule whose condition 
matches the current situation (as defined by the percept and the stored internal state) and then 
doing the action associated with that rule. 



Notice that decision-making of this kind is fundamentally different from the condition- 
action rules described earlier, in that it involves consideration of the future — both "What will 
happen if I do such-and-such?" and "Will that make me happy?" In the reflex agent designs, 
this information is not explicitly used, because the designer has precomputed the correct action 
for various cases. The reflex agent brakes when it sees brake lights. A goal-based agent, in 
principle, could reason that if the car in front has its brake lights on, it will slow down. From 
the way the world usually evolves, the only action that will achieve the goal of not hitting other 
cars is to brake. Although the goal-based agent appears less efficient, it is far more flexible. If it 
starts to rain, the agent can update its knowledge of how effectively its brakes will operate; this 
will automatically cause all of the relevant behaviors to be altered to suit the new conditions. For 
the reflex agent, on the other hand, we would have to rewrite a large number of condition-action 
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rules. Of course, the goal-based agent is also more flexible with respect to reaching different 
destinations. Simply by specifying a new destination, we can get the goal-based agent to come 
up with a new behavior. The reflex agent's rules for when to turn and when to go straight will 
only work for a single destination; they must all be replaced to go somewhere new. 

Figure 2.11 shows the goal-based agent's structure. Chapter 13 contains detailed agent 
programs for goal-based agents. 













Figure 2.11 An agent with explicit goals. 



Utility-based agents 

Goals alone are not really enough to generate high-quality behavior. For example, there are many 
action sequences that will get the taxi to its destination, thereby achieving the goal, but some I 
are quicker, safer, more reliable, or cheaper than others. Goals just provide a crude distinction 
between "happy" and "unhappy" states, whereas a more general performance measure should j 
allow a comparison of different world states (or sequences of states) according to exactly how ! 
happy they would make the agent if they could be achieved. Because "happy" does not sound I 
very scientific, the customary terminology is to say that if one world state is preferred to another, ] 
utility then it has higher utility for the agent. 8 

Utility is therefore a function that maps a state 9 onto a real number, which describes the j 
associated degree of happiness. A complete specification of the utility function allows rational j 
decisions in two kinds of cases where goals have trouble. First, when there are conflicting goals, 1 
only some of which can be achieved (for example, speed and safety), the utility function specifies 
the appropriate trade-off. Second, when there are several goals that the agent can aim for, none 

8 The word "utility" here refers to "the quality of being useful," not to the electric company or water works. 

9 Or sequence of states, if we are measuring the utility of an agent over the long run. 
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of which can be achieved with certainty, utility provides a way in which the likelihood of success 
can be weighed up against the importance of the goals. 

In Chapter 16, we show that any rational agent can be described as possessing a utility 
function. An agent that possesses an explicit utility function therefore can make rational decisions, 
but may have to compare the utilities achieved by different courses of actions. Goals, although 
cruder, enable the agent to pick an action right away if it satisfies the goal. In some cases, 
moreover, a utility function can be translated into a set of goals, such that the decisions made by 
a goal-based agent using those goals are identical to those made by the utility-based agent. 

The overall utility-based agent structure appears in Figure 2.12. Actual utility-based agent 
programs appeal in Chapter 5, where we examine game-playing programs that must make fine 
distinctions amoag various board positions; and in Chapter 17, where we tackle the general 
problem of designing decision-making agents. 




Z A Environments 



In this section and in the exercises at the end of the chapter, you will see how to couple an agent 
to an environment. Section 2.3 introduced several different kinds of agents and environments. 
In all cases, however, the nature of the connection between them is the same: actions are done 
by the agent on tie environment, which in turn provides percepts to the agent. First, we will 
describe the different types of environments and how they affect the design of agents. Then we 
will describe environment programs that can be used as testbeds for agent programs. 
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EPISODIC 



STATIC 



SEMIDYNAMIC 

DISCRETE 



Properties of environments 

Environments come in several flavors. The principal distinctions to be made are as follows: 

0 Accessible vs. inaccessible. 

If an agent's sensory apparatus gives it access to the complete state of the environment, 
then we say that the environment is accessible to that agent. An environment is effectively 
accessible if the sensors detect all aspects that are relevant to the choice of action. An 
accessible environment is convenient because the agent need not maintain any internal state 
to keep track of the world. 

0 Deterministic vs. nondeterministic. 

If the next state of the environment is completely determined by the current state and the 
actions selected by the agents, then we say the environment is deterministic. In principle, 
an agent need not worry about uncertainty in an accessible, deterministic environment. If 
the environment is inaccessible, however, then it may appear to be nondeterministic. This 
is particularly true if the environment is complex, making it hard to keep track of all the 
inaccessible aspects. Thus, it is often better to think of an environment as deterministic or 
nondeterministic/rom the point of view of the agent. 

0 Episodic vs. nonepisodic. 

In an episodic environment, the agent's experience is divided into "episodes." Each episode 
consists of the agent perceiving and then acting. The quality of its action depends just on 
the episode itself, because subsequent episodes do not depend on what actions occur in 
previous episodes. Episodic environments are much simpler because the agent does not 
need to think ahead. 

0 Static vs. dynamic. 

If the environment can change while an agent is deliberating, then we say the environment 
is dynamic for that agent; otherwise it is static. Static environments are easy to deal with 
because the agent need not keep looking at the world while it is deciding on an action, 
nor need it worry about the passage of time. If the environment does not change with the 
passage of time but the agent's performance score does, then we say the environment is 
semidynamic. 

0 Discrete vs. continuous. 

If there are a limited number of distinct, clearly defined percepts and actions we say that 
the environment is discrete. Chess is discrete — there are a fixed number of possible moves 
on each turn. Taxi driving is continuous — the speed and location of the taxi and the other 
vehicles sweep through a range of continuous values. 10 
We will see that different environment types require somewhat different agent programs to deal 
with them effectively. It will turn out, as you might expect, that the hardest case is inaccessible, 
nonepisodic, dynamic, and continuous. It also turns out that most real situations are so complex 
that whether they are really deterministic is a moot point; for practical purposes, they must be 
treated as nondeterministic. 

10 At a fine enough level of granularity, even the taxi driving environment is discrete, because the camera image is 
digitized to yield discrete pixel values. But any sensible agent program would have to abstract above this level, up to a 
level of granularity that is continuous. 
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Section 2.4. 

Figure 2.13 lists the properties of a number of familiar environments. Note that the answers 
can change depending on how you conceptualize the environments and agents. For example, 
poker is deterministic if the agent can keep track of the order of cards in the deck, but it is 
nondeterministic if it cannot. Also, many environments are episodic at higher levels than the 
agent's individual actions. For example, a chess tournament consists of a sequence of games; 
each game is an episode, because (by and large) the contribution of the moves in one game to the 
agent's overall performance is not affected by the moves in its next game. On the other hand, 
moves within a single game certainly interact, so the agent needs to look ahead several moves. 




Environment programs 

The generic environment program in Figure 2.14 illustrates the basic relationship between agents 
and environments. In this book, we will find it convenient for many of the examples and exercises 
to use an environment simulator that follows this program structure. The simulator takes one or 
more agents as input and arranges to repeatedly give each agent the right percepts and receive back 
an action. The simulator then updates the environment based on the actions, and possibly other 
dynamic processes in the environment that are not considered to be agents (rain, for example). 
The environment is therefore defined by the initial state and the update function. Of course, an 
agent that works in a simulator ought also to work in a real environment that provides the same 
kinds of percepts and accepts the same kinds of actions. 

The RUN-ENVIRONMENT procedure correctly exercises the agents in an environment. For 
some kinds of agents, such as those that engage in natural language dialogue, it may be sufficient 
simply to observe their behavior. To get more detailed information about agent performance, we 
insert some performance measurement code. The function RuN-EVAL-ENVIRONMENT, shown in 
Figure 2.15, does this; it applies a performance measure to each agent and returns a list of the 
resulting scores. The scores variable keeps track of each agent's score. 

In general, the performance measure can depend on the entire sequence of environment 
states generated during the operation of the program. Usually, however, the performance measure 
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procedure RuN-ENV[RONMENT(.v?«re, UPDATE-FN, agents, termination) 
inputs: state, the initial state of the environment 

UPDATE-FN, function to modify the environment 
agents, a set of agents 

termination, a predicate to test when we are done 

repeat 

for each agent in agents do 

PERCEPT[agent] — GEl-PERCEF\'(agent,state) 

end 

for each agent in agents do 

ACTlONjagent] — PROGRAM[agent](PERCEPT[agenf]) 

end 

state <— UPDATE-FN(acfi0n.s, agents, state) 
until termination) state) 



Figure 2.14 The basic environment simulator program. It gives each agent its percept, gets an 
action from each agent, and then updates the environment. 



function Run-Eval-Environment( .stare, UPDATE-FN, agents, 

termination, PERFORMANCE-FN) returns scores 
local variables: scores, a vector the same size as agents, all 0 

repeat 

for each agent in agents do 

PERCEPTfagent] <— GET-PERCEPT(agenf, state) 

end 

for each agent in agents do 

AcnON[agenf] — PROGRAM[agent](PERCEPT[agenf]) 

end 

state <— UPDATE-FN(acftonj, agents, state) 
scores ■ — PERFORMANCE-FNfrcore.?, agents, state) 
until termination{state ) 

return scores I * change * / 



Figure 2.15 An environment simulator program that keeps track of the performance measure 
for each agent. 



works by a simple accumulation using either summation, averaging, or taking a maximum. For 
example, if the performance measure for a vacuum-cleaning agent is the total amount of dirt 
cleaned in a shift, scores will just keep track of how much dirt has been cleaned up so far. 

Run-Eval-Environment returns the performance measure for a a single environment, 
defined by a single initial state and a particular update function. Usually, an agent is designed to 
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5lass° NM£NT wor ^ ' n an environment class, a whole set of different environments. For example, we design 
a chess program to play against any of a wide collection of human and machine opponents. If 
we designed it for a single opponent, we might be able to take advantage of specific weaknesses 
in that opponent, but that would not give us a good program for general play. Strictly speaking, 
in order to measure the performance of an agent, we need to have an environment generator 
that selects particular environments (with certain likelihoods) in which to run the agent. We are 
then interested in the agent's average performance over the environment class. This is fairly 
straightforward to implement for a simulated environment, and Exercises 2.5 to 2.1 1 take you 
through the entire development of an environment and the associated measurement process. 

A possible confusion arises between the state variable in the environment simulator and 
the state variable in the agent itself (see Reflex- Agent- With-State). As a programmer imple- 
menting both the environment simulator and the agent, it is tempting to allow the agent to peek 
at the environment simulator's state variable. This temptation must be resisted at all costs! The 
agent's version of the state must be constructed from its percepts alone, without access to the 
complete state information. 



2.5 SUMMARY 



This chapter has been something of a whirlwind tour of AI, which we have conceived of as the 
science of agent design. The major points to recall are as follows: 

• An agent is something that perceives and acts in an environment. We split an agent into 
an architecture and an agent program. 

• An ideal agent is one that always takes the action that is expected to maximize its perfor- 
mance measure, given the percept sequence it has seen so far. 

• An agent is autonomous to the extent that its action choices depend on its own experience, 
rather than on knowledge of the environment that has been built-in by the designer. 

• An agent program maps from a percept to an action, while updating an internal state. 

• There exists a variety of basic agent program designs, depending on the kind of information 
made explicit and used in the decision process. The designs vary in efficiency, compactness, 
and flexibility. The appropriate design of the agent program depends on the percepts, 
actions, goals, and environment. 

• Reflex agents respond immediately to percepts, goal-based agents act so that they will 
achieve their goal(s), and utility-based agents try to maximize their own "happiness." 

• The process of making decisions by reasoning with knowledge is central to AI and to 
successful agent design. This means that representing knowledge is important. 

• Some environments are more demanding than others. Environments that are inaccessible, 
nondeterministic, nonepisodic, dynamic, and continuous are the most challenging. 
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BIBLIOGRAPHICAL AND HISTORICAL NOTES 

The analysis of rational agency as a mapping from percept sequences to actions probably stems 
ultimately from the effort to identify rational behavior in the realm of economics and other forms 
of reasoning under uncertainty (covered in later chapters) and from the efforts of psychological 
behaviorists such as Skinner (195 3) to reduce the psychology of organisms strictly to input/output 
or stimulus/response mappings. The advance from behaviorism to functionalism in psychology, 
which was at least partly driven by the application of the computer metaphor to agents (Putnam, 
1960; Lewis, 1 966), introduced the internal state of the agent into the picture. The philosopher 
Daniel Dennett (1969; 1978b) helped to synthesize these viewpoints into a coherent "intentional 
stance" toward agents. A high-level, abstract perspective on agency is also taken within the world 
of AI in (McCarthy and Hayes, 1969). Jon Doyle (1983) proposed that rational agent design is 
the core of AI, and would remain as its mission while other topics in AI would spin off to form 
new disciplines. Horvitz et al. (1988) specifically suggest the use of rationality conceived as the 
maximization of expected utility as a basis for AI. 

The AI researcher and Nobel-prize-winning economist Herb Simon drew a clear distinction 
between rationality under resource limitations (procedural rationality) and rationality as making 
the objectively rational choice (substantive rationality) (Simon, 1958). Cherniak (1986) explores 
the minimal level of rationality needed to qualify an entity as an agent. Russell and Wefald (1991) 
deal explicitly with the possibility of using a variety of agent architectures. Dung Beetle Ecol- 
ogy (Hanski and Cambefort, 1991) provides a wealth of interesting information on the behavior 
of dung beetles. 



EXERCISES 



2.1 What is the difference between a performance measure and a utility function? 

2.2 For each of the environments in Figure 2.3, determine what type of agent architecture is 
most appropriate (table lookup, simple reflex, goal-based or utility-based). 

2.3 Choose a domain that you are familiar with, and write a PAGE description of an agent 
for the environment. Characterize the environment as being accessible, deterministic, episodic, 
static, and continuous or not. What agent architecture is best for this domain? 

2.4 While driving, which is the best policy? 

a. Always put your directional blinker on before turning, 

b. Never use your blinker, 

c. Look in your mirrors and use your blinker only if you observe a car that can observe you? 
What kind of reasoning did you need to do to arrive at this policy (logical, goal-based, or utility- 
based)? What kind of agent design is necessary to carry out the policy (reflex, goal-based, or 
utility-based)? 
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Section 2.5. 



The following exercises all concern the implementation of an environment and set of agents in 
the vacuum-cleaner world. 

2.5 Implement a performance-measuring environment simulator for the vacuum-cleaner world. 
This world can be described as follows: 

0 Percepts: Each vacuum-cleaner agent gets a three-element percept vector on each turn. 
The first element, a touch sensor, should be a 1 if the machine has bumped into something 
and a 0 otherwise. The second comes from a photosensor under the machine, which emits 
a 1 if there is dirt there and a 0 otherwise. The third comes from an infrared sensor, which 
emits a 1 when the agent is in its home location, and a 0 otherwise. 

0 Actions: There are five actions available: go forward, turn right by 90° , turn left by 90° , 
suck up dirt, and turn off. 

0 Goals: The goal for each agent is to clean up and go home. To be precise, the performance 
measure will be 100 points for each piece of dirt vacuumed up, minus 1 point for each 
action taken, and minus 1000 points if it is not in the home location when it turns itself off. 

O Environment: The environment consists of a grid of squares. Some squares contain 
obstacles (walls and furniture) and other squares are open space. Some of the open squares 
contain dirt. Each "go forward" action moves one square unless there is an obstacle in that 
square, in which case the agent stays where it is, but the touch sensor goes on. A "suck up 
dirt" action always cleans up the dirt. A "turn off" command ends the simulation. 

We can vary the complexity of the environment along three dimensions: 

O Room shape: In the simplest case, the room is an n x n square, for some fixed n. We can 
make it more difficult by changing to a rectangular, L-shaped, or irregularly shaped room, 
or a series of rooms connected by corridors. 

0 Furniture: Placing furniture in the room makes it more complex than an empty room. To 
the vacuum-cleaning agent, a piece of furniture cannot be distinguished from a wall by 
perception; both appear as a 1 on the touch sensor. 

0 Dirt placement: In the simplest case, dirt is distributed uniformly around the room. But 
it is more realistic for the dirt to predominate in certain locations, such as along a heavily 
travelled path to the next room, or in front of the couch. 

2.6 Implement a table-lookup agent for the special case of the vacuum-cleaner world consisting 
of a 2 x 2 grid of open squares, in which at most two squares will contain dirt. The agent starts 
in the upper left comer, facing to the right. Recall that a table-lookup agent consists of a table of 
actions indexed by a percept sequence. In this environment, the agent can always complete its 
task in nine or fewer actions (four moves, three turns, and two suck-ups), so the table only needs 
entries for percept sequences up to length nine. At each turn, there are eight possible percept 
vectors, so the table will be of size 8 9 = 134,217,728. Fortunately, we can cut this down by 
realizing that the touch sensor and home sensor inputs are not needed; we can arrange so that 
the agent never bumps into a wall and knows when it has returned home. Then there are only 
two relevant percept vectors, ?0? and ?1?, and the size of the table is at most 2 9 = 512. Run the 
environment simulator on the table-lookup agent in all possible worlds (how many are there?). 
Record its performance score for each world and its overall average score. 
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2.7 Implement an environment for a nxm rectangular room, where each square has a 5% chance 
of containing dirt, and n and m are chosen at random from the range 8 to 15, inclusive. 

2.8 Design and implement a pure reflex agent for the environment of Exercise 2.7, ignoring 
the requirement of returning home, and measure its performance. Explain why it is impossible 
to have a reflex agent that returns home' and shuts itself off. Speculate on what the best possible 
reflex agent could do. What prevents a reflex agent from doing very well? 

2.9 Design and implement several agents with internal state. Measure their performance. How 
close do they come to the ideal agent for this environment? 

2.10 Calculate the size of the table for a table-lookup agent in the domain of Exercise 2.7. 
Explain your calculation. You need not fill in the entries for the table. 

211 Experiment with changing the shape and dirt placement of the room, and with adding 
furniture. Measure your agents in these new environments. Discuss how their performance 
might be improved to handle more complex geographies. 



Part II 



PROBLEM-SOLVING 



In this part we show how an agent can act by establishing goals and considering 
sequences of actions that might achieve those goals. A goal and a set of means 
for achieving the goal is called a problem, and the process of exploring what the 
means can do is called search. We show what search can do, how it must be 
modified to account for adversaries, and what its limitations are. 



1 





SOLVING PROBLEMS BY 
SEARCHING 



In which we look at how an agent can decide what to do by systematically considering 
the outcomes ofvarious sequences of actions that it might take. 



In Chapter 2, we saw that simple reflex agents are unable to plan ahead. They are limited in what 
they can do because their actions are determined only by the current percept. Furthermore, they 
have no knowledge of what their actions do nor of what they are trying to achieve. 
probl e m-s°lvin g In this chapter, we describe one kind of goal-based agent called a problem-solving agent. 

Problem-solving agents decide what to do by finding sequences of actions that lead to desirable 
states. We discuss informally how the agent can formulate an appropriate view of the problem it 
faces. The problem type that results from the formulation process will depend on the knowledge 
available to the agent: principally, whether it knows the current state and the outcomes of actions. 
We then define more precisely the elements that constitute a "problem" and its "solution," and 
give several examples to illustrate these definitions. Given precise definitions of problems, it 
is relatively straightforward to construct a search process for finding solutions. We cover six 
different search strategies and show how they can be applied to a variety of problems. Chapter 4 
will then cover search strategies that make use of more information about the problem to improve 
the efficiency of the search process. 

This chapter uses concepts from the analysis of algorithms. Readers unfamiliar with the 
concepts of asymptotic complexity and NP-completeness should consult Appendix A. 



Problem-Solving agents 



Intelligent agents are supposed to act in such a way that the environment goes through a sequence 
of states that maximizes the performance measure. In its full generality, this specification is 
difficult to translate into a successful agent design. As we mentioned in Chapter 2, the task is 
somewhat simplified if the agent can adopt a goal and aim to satisfy it. Let us first look at how 
and why an agent might do this. 
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Imagine our agent in the city of Arad, Romania, toward the end of a touring holiday. The 
agent has a ticket to fly out of Bucharest the following day. The ticket is nonrefundable, the 
agent's visa is about to expire, and after tomorrow, there are no seats available for six weeks. Now 
the agent's performance measure contains many other factors besides the cost of the ticket and 
the. undesirability of being arrested aqd deported. For example, it wants to improve its suntan, 
improve its Romanian, take in the sights, and so on. All these factors might suggest any of a vast 
array of possible actions. Given the seriousness of the situation, however, it should adopt the 
goal of driving to Bucharest. Actions that result in a failure to reach Bucharest on time can be 
rejected without further consideration. Goals such as this help organize behavior by limiting the 
goal formulation objectives that the agent is trying to achieve. Goal formulation, based on the current situation, 
is the first step in problem solving. As well as formulating a goal, the agent may wish to decide 
on some other factors that affect the desirability of different ways of achieving the goal. 

For the purposes of this chapter, we will consider a goal to be a set of world states — just 
those states in which the goal is satisfied. Actions can be viewed as causing transitions between 
world states, so obviously the agent has to find out which actions will get it to a goal state. Before 
it can do this, it needs to decide what sorts of actions and states to consider. If it were to try 
to consider actions at the level of "move the left foot forward 18 inches" or "turn the steering 
wheel six degrees left," it would never find its way out of the parking lot, let alone to Bucharest, 
because constructing a solution at that level of detail would be an intractable problem. Problem 
formulation formulation is the process of deciding what actions and states to consider, and follows goal 
formulation. We will discuss this process in more detail. For now, let us assume that the agent 
will consider actions at the level of driving from one major town to another. The states it will 
consider therefore correspond to being in a particular town. 1 

Our agent has now adopted the goal of driving to Bucharest, and is considering which town 
to drive to from Arad. There are three roads out of Arad, one toward Sibiu, one to Timisoara, 
and one to Zerind. None of these achieves the goal, so unless the agent is very familiar with the 
geography of Romania, it will not know which road to follow. 2 In other words, the agent will not ] 
know which of its possible actions is best, because it does not know enough about the state that j 
results from taking each action. If the agent has no additional knowledge, then it is stuck. The i 
best it can do is choose one of the actions at random. 

But suppose the agent has a map of Romania, either on paper or in its memory. The point 1 
of a map is to provide the agent with information about the states it might get itself into, and j 
the actions it can take. The agent can use this information to consider subsequent stages of a j 
hypothetical journey through each of the three towns, to try to find a journey that eventually gets j 
to Bucharest. Once it has found a path on the map from Arad to Bucharest, it can achieve its goal 
by carrying out the driving actions that correspond to the legs of thejoumey. In general, then, an j 
agent with several immediate options of unknown value can decide what to do by first examining 
different possible sequences of actions that lead to states of known value, and then choosing the j 
search best one. This process of looking for such a sequence is called search. A search algorithm takes 

solution a problem as input and returns a solution in the form of an action sequence. Once a solution is 

1 Notice that these states actually correspond to large sets of world states, because a world state specifies every aspect 
of reality. It is important to keep in mind the distinction between states in problem solving and world states. 

2 We are assuming that most readers are in the same position, and can easily imagine themselves as clueless as our 
agent. We apologize to Romanian readers who are unable to take advantage of this pedagogical device. 
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Formulating Problems 



execution found, the actions it recommends can be carried out. This is called the execution phase. Thus, 

we have a simple "formulate, search, execute" design for the agent, as shown in Figure 3.1. After 
formulating a goal and a problem to solve, the agent calls a search procedure to solve it. It then 
uses the solution to guide its actions, doing whatever the solution recommends as the next thing 
to do, and then removing that step from the sequence. Once the solution has been executed, the 
agent will find a new goal. 



function Simple-Problem-Solving-Agenti p ) returns an action 
inputs: p, a percept 

static: s, an action sequence, initially empty 

state, some description of the current world state 
g, a goal, initially null 
problem, a problem formulation 

state — U pdate-State(5?<2/<?, p ) 
if s is empty then 

g — Form ulate-GoalU tore) 
problem — FORMULATE-PROBLEMtoMto, g) 
s »s- SEARCHtproMem) 
action — RECOMMENDATION^-, state) 
s «- Remainder^, state) 
return action 



Figure 3.1 A simple problem-solving agent. 



We will not discuss the UPDATE-STATE and Formulate-Goal functions further in this 
chapter. The next two sections describe the process of problem formulation, and then the 
remainder of the chapter is devoted to various versions of the SEARCH function. The execution 
phase is usually straightforward for a simple problem-solving agent: RECOMMENDATION just 
takes the first action in the sequence, and REMAINDER returns the rest. 



3.2 FORMULATING PROBLEMS 



In this section, we will consider the problem formulation process in more detail. First, we will 
look at the different amounts of knowledge that an agent can have concerning its actions and the 
state that it is in. This depends on how the agent is connected to its environment through its 
percepts and actions. We find that there are four essentially different types of problems — single- 
state problems, multiple-state problems, contingency problems, and exploration problems. We 
will define these types precisely, in preparation for later sections that address the solution process. 
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Knowledge and problem types 

Let us consider an environment somewhat different from Romania: the vacuum world from 
Exercises 2.5 to 2.1 1 in Chapter 2. We will simplify it even furtherfor the sake of exposition. Let 
the world contain just two locations. Each location may or may not contain dirt, and the agent 
may be in one location or the other. There are 8 possible world states, as shown in Figure 3.2. 
The agent has three possible actions in this version of the vacuum world: Left.Right, and Suck. 
Assume, for the moment, that sucking is 100% effective. The goal is to clean up all the dirt. That 
is, the goal is equivalent to the state set {7, 8). 
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Figure 3.2 The eight possible states of the simplified vacuum world. 



MULTIPLE-STATE 



First, suppose that the agent's sensors give it enough information to tell exactly which state 
it is in (i.e., the world is accessible); and suppose that it knows exactly what each of its actions 
does. Then it can calculate exactly which state it will be in after any sequence of actions. For 
example, if its initial state is 5, then it can calculate that the action sequence [Right, Suck^iW get 
to a goal state. This is the simplest case, which we call a single-state problem. 

Second, suppose that the agent knows all the effects of its actions, but has limited access 
to the world state. For example, in the extreme case, it may have no sensors at all. In that case, 
it knows only that its initial state is one of the set { 1, 2, 3, 4, 5, 6, 7, 8}. One might suppose that 
the agent's predicament is hopeless, but in fact it can do quite well. Because it knows what its 
actions do, it can, for example, calculate that the action Right will cause it to be in one of the 
states {2,4, 6, 8). In fact, the agent can discover that the action sequence \ Right. Suck.Left, Suck] 
is guaranteed to reach a goal state no matter what the start state. To summarize: when the world 
is not fully accessible, the agent must reason about sets of states that it might get to, rather than 
single states. We call this a multiple-state problem. 
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Although it might seem different, the case of ignorance about the effects of actions can be 
treated similarly. Suppose, for example, that the environment appears to be nondeterministic in 
that it obeys Murphy's Law: the so-called Suck action sometimes deposits dirt on the carpet but 
only if there is no dirt there already} For example, if the agent knows it is in state 4, then it 
knows that if it sucks, it will reach me of the states {2,4}. For any known initial state, however, 
there is an action sequence that is guaranteed to reach a goal state (see Exercise 3.2). 

Sometimes ignorance prevents the agent from finding a guaranteed solution sequence. 
Suppose, for example, that the agent is in the Murphy's Law world, and that it has a position 
sensor and a local dirt sensor, but no sensor capable of detecting dirt in other squares. Suppose 
further that the sensors tell it that it is in one of the states {1,3}. The agent might formulate the 
action sequence [Suck, Right, Suck], Sucking would change the state to one of {5, 7}, and moving 
right would then change the state to one of {6, 8}. If it is in fact state 6, then the action sequence 
will succeed, but if it is state 8, the plan will fail. If the agent had chosen the simpler action 
sequence [Suck], it would also succeed some of the time, but not always. It turns out there is no 
fixed action sequence that guarantees a solution to this problem. 

Obviously, the agent does have a way to solve the problem starting from one of { 1 , 3} : first 
suck, then move right, then suck only if there is dirt there. Thus, solving this problem requires 
sensing during the execution phase. Notice that the agent must now calculate a whole tree of 
actions, rather than a single action sequence. In general, each branch of the tree deals with a 
contingency possible contingency that might arise. For this reason, we call this a contingency problem. 

Many problems in the real, physical world are contingency problems, because exact prediction is 
impossible. For this reason, many people keep their eyes open while walking around or driving. 

Single-state and multiple-state problems can be handled by similar search techniques, 
which are covered in this chapter and the next. Contingency problems, on the other hand, 
require more complex algorithms, which we cover in Chapter 13. They also lend themselves to a 
somewhat different agent design, in which the agent can act before it has found a guaranteed plan. 
This is useful because rather than considering in advance every possible contingency that might 
arise during execution, it is often better to actually start executing and see which contingencies 
do arise. The agent can then continue to solve the problem given the additional information. This 
interleaving type of interleaving of search and execution is also covered in Chapter 13, and for the limited 
case of two-player games, in Chapter 5. For the remainder of this chapter, we will only consider 
cases where guaranteed solutions consist of a single sequence of actions. 

Finally, consider the plight of an agent that has no information about the effects of its 
actions. This is somewhat equivalent to being lost in a strange country with no map at all, and is 
the hardest task faced by an intelligent agent. 4 The agent must experiment, gradually discovering 
what its actions do and what sorts of states exist. This is a kind of search, but a search in the 
real world rather than in a model thereof. Taking a step in the real world, rather than in a model, 
may involve significant danger for an ignorant agent. If it survives, the agent learns a "map" of 
the environment, which it can then use to solve subsequent problems. We discuss this kind of 
BBSi™ exploration problem in Chapter 20. 



3 We assume that most readers face similar problems, and can imagine themselves as frustrated as our agent. We 

4 It is also the task faced by newborn babies. 

L 
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PROBLEM 



INITIAL STATE 



OPERATOR 



GOALTEST 



SOLUTION 



Well-defined problems and solutions 

A problem is really a collection of information that the agent will use to decide what to do. We 
will begin by specifying the information needed to define a single-state problem. 

We have seen that the basic elements of a problem definition are the states and actions. To 
capture these formally, we need the following: 

• , The initial state that the agent knows itself to be in. 

• The set of possible actions available to the agent. The term operator is used to denote 
the description of an action in terms of which state will be reached by carrying out the 
action in a particular state. (An alternate formulation uses a successor function S. Given 
a particular state x, S(x) returns the set of states reachable from x by any single action.) 

Together, these define the state space of the problem: the set of all states reachable from the 
initial state by any sequence of actions. A path in the state space is simply any sequence of 
actions leading from one state to another. The next element of a problem is the following: 

• The goal test, which the agent can apply to a single state description to determine if it is 
a goal state. Sometimes there is an explicit set of possible goal states, and the test simply 
checks to see if we have reached one of them. Sometimes the goal is specified by an 
abstract property rather than an explicitly enumerated set of states. For example, in chess, 
the goal is to reach a state called "checkmate,'' where the opponent's king can be captured 
on the next move no matter what the opponent does. 

Finally, it may be the case that one solution is preferable to another, even though they both reach 
the goal. For example, we might prefer paths with fewer or less costly actions. 

• A path cost function is a function that assigns a cost to a path. In all cases we will consider, 
the cost of a path is the sum of the costs of the individual actions along the path. The path 
cost function is often denoted by g. 

Together, the initial state, operator set, goal test, and path cost function define a problem. 
Naturally, we can then define a datatype with which to represent problems: 



datatype PROBLEM 

components: INITIAL-STATE, OPERATORS, GOAL-TEST, PATH-COST-FUNCTION 



Instances of this datatype will be the input to our search algorithms. The output of a search 
algorithm is a solution, that is, a path from the initial state to a state that satisfies the goal test. 

To deal with multiple-state problems, we need to make only minor modifications: a problem 
consists of an initial state set', a set of operators specifying for each action the set of states reached 
from any given state; and a goal test and path cost function as before. An operator is applied to 
a state set by unioning the results of applying the operator to each state in the set. A path now 
connects sets of states, and a solution is now a path that leads to a set of states all of which are 
goal states. The state space is replaced by the state set space (see Figure 3.7 for an example). 
Problems of both types are illustrated in Section 3.3. 




Section 3.2. 



Formula' 



Problems 



Measuring problem-solving performance 

The effectiveness of a search can be measured in at least three ways. First, does it rind a solution 
search COST at all? Second, is it a good solution (one with a low path cost)? Third, what is the search cost 

total COST associated with the time and memory required to find a solution? The total cost of the search is 

the sum of the path cost and the search cost. 5 

For the problem of finding a route from Arad to Bucharest, the path cost might be pro- 
portional to the total mileage of the path, perhaps with something thrown in for wear and tear 
on different road surfaces. The search cost will depend on the circumstances. In a static en- 
vironment, it will be zero because the performance measure is independent of time. If there is 
some urgency to get to Bucharest, the environment is semidynamic because deliberating longer 
will cost more. In this case, the search cost might vary approximately linearly with computation 
time (at least for small amounts of time). Thus, to compute the total cost, it would appear that 
we have to add miles and milliseconds. This is not always easy, because there is no "official 
exchange rate" between the two. The agent must somehow decide what resources to devote to 
search and what resources to devote to execution. For problems with very small state spaces, it 
is easy to find the solution with the lowest path cost. But for large, complicated problems, there 
is a trade-off to be made — the agent can search for a very long time to get an optimal solution, 
or the agent can search for a shorter time and get a solution with a slightly larger path cost. The 
issue of allocating resources will be taken up again in Chapter 16; for now, we concentrate on 
the search itself. 



Choosing states and actions 

Now that we have the definitions out of the way, let us start our investigation of problems with 
an easy one: "Drive from Arad to Bucharest using the roads in the map in Figure 3.3." An 
appropriate state space has 20 states, where each state is defined solely by location, specified as 
a city. Thus, the initial state is "in Arad” and the goal test is "is this Bucharest?" The operators 
correspond to driving along the roads between cities. 

One solution is the path Arad to Sibiu to Rimnicu Vilcea to Pitesti to Bucharest. There are 
lots of other paths that are also solutions, for example, via Lugoj and Craiova. To decide which 
of these solutions is better, we need to know what the path cost function is measuring: it could 
be the total mileage, or the expected travel time. Because our current map does not specify either 
of these, we will use the number of steps as the cost function. That means that the path through 
Sibiu and Fagaras, with a path cost of 3, is the best possible solution. 

The real art of problem solving is in deciding what goes into the description of the states 
and operators and what is left out. Compare the simple state description we have chosen, "in 
Arad," to an actual cross-country trip, where the state of the world includes so many things: the 
travelling companions, what is on the radio, what there is to look at out of the window, the vehicle 
being used for the trip, how fast it is going, whether there are any law enforcement officers nearby, 
what time it is, whether the driver is hungry or tired or running out of gas, how far it is to the next 

5 In theoretical computer science and in robotics, the search cost (the part you do before interacting with the environment) 
is called the offline cost and the path cost is called the online cost. 
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rest stop, the condition of the road, the weather, and so on. All these considerations are left out 
of state descriptions because they are irrelevant to the problem of finding a route to Bucharest. 

ABSTRACTION The process of removing detail from a representation is called abstraction. 

As well as abstracting the state description, we must abstract the actions themselves. An 
action — let us say a car trip from Arad to Zerind — has many effects. Besides changing the 
location of the vehicle and its occupants, it takes up time, consumes fuel, generates pollution, and 
changes the agent (as they say, travel is broadening). In our formulation, we take into account 
only the change in location. Also, there are many actions that we will omit altogether: turning j 
on the radio, looking out of the window, slowing down for law enforcement officers, and so on. 

Can we be more precise about defining the appropriate level of abstraction? Think of the 
states and actions we have chosen as corresponding to sets of detailed world states and sets of 
detailed action sequences. Now consider a solution to the abstract problem: for example, the _ 
path Arad to Sibiu to Rimnicu Vilcea to Pitesti to Bucharest. This solution corresponds to a large 
number of more detailed paths. For example, we could drive with the radio on between Sibiu 
and Rimnicu Vilcea, and then switch it off for the rest of the trip. Each of these more detailed 
paths is still a solution to the goal, so the abstraction is valid. The abstraction is also useful, 
because carrying out each of the actions in the solution, such as driving from Pitesti to Bucharest, 
is somewhat easier than the original problem. The choice of a good abstraction thus involves 
removing as much detail as possible while retaining validity and ensuring that the abstract actions 
are easy to carry out. Were it not for the ability to construct useful abstractions, intelligent agents 
would be completely swamped by the real world. 
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Example PROBLEMS. 



The range of task environments that can be characterized by well-defined problems is vast. We 
T0Y problems can distinguish between so-called, toy problems, which are intended to illustrate or exercise 

Problems 10 various problem-solving methods, and so-called real-world problems, which tend to be more 

difficult and whose solutions people actually care about. In this section, we will give examples of 
both. By nature, toy problems can be given a concise, exact description. This means that they can 
be easily used by different researchers to compare the performance of algorithms. Real-world 
problems, on the other hand, tend not to have a single agreed-upon description, but we will 
attempt to give the general flavor of their formulations. 




Toy problems 

The 8-puzzle 

The 8-puzzle, an instance of which is shown in Figure 3.4, consists of a 3x3 board with eight 
numbered tiles and a blank space. A tile adjacent to the blank space can slide into the space. 
The object is to reach the configuration shown on the right of the figure. One important trick is 
to notice that rather than use operators such as "move the 3 tile into the blank space," it is more 
sensible to have operators such as "the blank space changes places with the tile to its left." This is 
because there are fewer of the latter kind of operator. This leads us to the following formulation: 
0 States: a state description specifies the location of each of the eight tiles in one of the nine 
squares. For efficiency, it is useful to include the location of the blank. 

0 Operators: blank moves left, right, up, or down. 

0 Goal test: state matches the goal configuration shown in Figure 3.4. 

0 Path cost: each step costs 1, so the path cost is just the length of the path. 

The 8-puzzle belongs to the family of sliding-block puzzles. This general class is known 
to be NP-complete, so one does not expect to find methods significantly better than the search 




Figure 3.4 A typical instance of the 8-puzzle. 
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algorithms described in this chapter and the next. The 8-puzzle and its larger cousin, the 15- 
puzzle, are the standard test problems for new search algorithms in Al. 

The 8-queens problem 

The goal of the 8-queens problem is to place eight queens on a chessboard such that no queen 
attacks any other. (A queen attacks any piece in the same row, column or diagonal.) Figure 3.5 
shows an attempted solution that fails: the queen in the rightmost column is attacked by the queen 
at top left. 









Figure 3.5 Almost a solution to the 8-queens problem. (Solution is left as an exercise.) 



Although efficient special-purpose algorithms exist for this problem and the whole n- 
queens family, it remains an interesting test problem for search algorithms. There are two main 
kinds of formulation. The incremental formulation involves placing queens one by one, whereas 
the complete-state formulation starts with all 8 queens on the board and moves them around. In 
either case, the path cost is of no interest because only the final state counts; algorithms are thus 
compared only on search cost. Thus, we have the following goal test and path cost: 

0 Goal test: 8 queens on board, none attacked. 

0 Path cost: zero. 

There are also different possible states and operators. Consider the following simple-minded 
formulation: 

<C> States: any arrangement of 0 to 8 queens on board. 

0 Operators: add a queen to any square. 

In this formulation, we have 64 8 possible sequences to investigate. A more sensible choice would 
use the fact that placing a queen where it is already attacked cannot work, because subsequent 
placings of other queens will not undo the attack. So we might try the following: 
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0 States: arrangements of 0 to 8 queens with none attacked. 

0 Operators: place a queen in the left-most empty column such that it is not attacked by any 
other queen. 

It is easy to see that the actions given can generate only states with no attacks; but sometimes 
no actions will be possible. For example, after making the first seven choices (left-to-right) in 
Figure 3.5, there is no action available in this formulation. The search process must try another 
choice. A quick calculation shows that there are only 2057 possible sequences to investigate. The 
rightformulation makes a big difference to the size of the search space. Similar considerations 
apply for a complete-state formulation. For example, we could set the problem up as follows: 

0 States: arrangements of 8 queens, one in each column. 

0 Operators: move any attacked queen to another square in the same column. 

This formulation would allow the algorithm to find a solution eventually, but it would be better 
to move to an unattacked square if possible. 



Cryptarithmetic 

In cryptarithmetic problems, letters stand for digits and the aim is to find a substitution of digits 
for letters such that the resulting sum is arithmetically correct. Usually, each letter must stand 
for a different digit. The following is a well-known example: 

FORTY Solution: 29786 F=2, 0=9, R=7, etc. 

+ TEN 850 

+ TEN 850 



SIXTY 31486 

The following formulation is probably the simplest: 

0 States: a cryptarithmetic puzzle with some letters replaced by digits. 

0 Operators: replace all occurrences of a letter with a digit not already appearing in the 
puzzle. 

0 Goal test: puzzle contains only digits, and represents a correct sum. 

0 Path cost: zero. All solutions equally valid. 

A moment's thought shows that replacing E by 6 then F by 7 is the same thing as replacing F by 
7 then E by 6 — order does not matter to correctness, so we want to avoid trying permutations of 
the same substitutions. One way to do this is to adopt a fixed order, e.g., alphabetical order. A 
better choice is to do whichever is the most constrained substitution, that is, the letter that has 
the fewest legal possibilities given the constraints of the puzzle. 



The vacuum world 

Here we will define the simplified vacuum world from Figure 3.2, rather than the full version 
from Chapter 2. The latter is dealt with in Exercise 3.17. 
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First, let us review the single-state case with complete information. We assume that the 
agent knows its location and the locations of all the pieces of dirt, and that the suction is still in 
good working order. 

O’ States: one of the eight states shown in Figure 3.2 (or Figure 3.6). 

0 Operators: move left, move 'right, suck. 

0 Goal test: no dirt left in any square. 

0 Path cost: each action costs 1 . 




Figure 3.6 shows the complete state space showing all the possible paths. Solving the ; 
problem from any starting state is simply a matter of following arrows to a goal state. This is the j 
case for all problems, of course, but in most, the state space is vastly larger and more tangled. 

Now let us consider the case where the agent has no sensors, but still has to clean up all 
the dirt. Because this is a multiple- state problem, we will have the following: 

0 State sets: subsets of states 1-8 shown in Figure 3.2 (or Figure 3.6). 

<C> Operators: move left, move right, suck. 

0 Goal test: all states in state set have no dirt. 

<C> Path cost: each action costs 1 . 

The start state set is the set of all states, because the agent has no sensors. A solution is any 
sequence leading from the start state set to a set of states with no dirt (see Figure 3.7). Similar 
state set spaces can be constructed for the case of uncertainty about actions and uncertainty about 
both states and actions. 
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Missionaries and cannibals 

The missionaries and cannibals problem is usually stated as follows. Three missionaries and 
three cannibals are on one side of a river, along with a boat that can hold one or two people. Find 
a way to get everyone to the other side, without ever leaving a group of missionaries in one place 
outnumbered by the cannibals in that place. 

This problem is famous in AI because it was the subject of the first paper that approached 
problem formulation from an analytical viewpoint (Amarel, 1968). As with travelling in Romania, 
the real-life problem must be greatly abstracted before we can apply a problem-solving strategy. 
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Imagine the scene in real life: three members of the Arawaskan tribe, Alice, Bob, and Charles, 
stand at the edge of the crocodile-infested Amazon river with their new-found friends, Xavier, 
Yolanda, and Zelda. All around them birds cry, a rain storm beats down, Tarzan yodels, and so 
on. The missionaries Xavier, Yolanda, and Zelda are a little worried about what might happen if 
one of them were caught alone with two or three of the others, and Alice, Bob, and Charles are 
concerned that they might be in for a long sermon that they might find equally unpleasant. Both 
parties are not quite sure if the small boat they find tied up by the side of the river is up to making 
the crossing with two aboard. 

To formalize the problem, the first step is to forget about the rain, the crocodiles, and all the 
other details that have no bearing in the solution. The next step is to decide what the right operator 
set is. We know that the operators will involve taking one or two people across the river in the 
boat, but we have to decide if we need a state to represent the time when they are in the boat, or 
just when they get to the other side. Because the boat holds only two people, no "outnumbering" 
can occur in it; hence, only the endpoints of the crossing are important. Next, we need to abstract 
over the individuals. Surely, each of the six is a unique human being, but for the purposes of the 
solution, when it comes time for a cannibal to get into the boat, it does not matter if it is Alice, 
Bob, or Charles. Any permutation of the three missionaries or the three cannibals leads to the 
same outcome. These considerations lead to the following formal definition of the problem: 

0 States: a state consists of an ordered sequence of three numbers representing the number 
of missionaries, cannibals, and boats on the bank of the river from which they started. 
Thus, the start state is (3,3,1). 

0 Operators: from each state the possible operators are to take either one missionary, one 
cannibal, two missionaries, two cannibals, or one of each across in the boat. Thus, there 
are at most five operators, although most states have fewer because it is necessary to avoid 
illegal states. Note that if we had chosen to distinguish between individual people then 
there would be 27 operators instead of just 5. 

0 Goal test: reached state (0,0,0). 

0 Path cost: number of crossings. 

This state space is small enough to make it a trivial problem for a computer to solve. People 
have a hard time, however, because some of the necessary moves appear retrograde. Presumably, J 
humans use some notion of "progress" to guide their search. We will see how such notions are 
used in the next chapter. 



Real-world problems 

Route finding 

We have already seen how route finding is defined in terms of specified locations and transitions 
along links between them. Route-finding algorithms are used in a variety of applications, such 
as routing in computer networks, automated travel advisory systems, and airline travel planning 
systems. The last application is somewhat more complicated, because airline travel has a very : 
complex path cost, in terms of money, seat quality, time of day, type of airplane, frequent-flyer 
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mileage awards, and so on. Furthermore, the actions in the problem do not have completely 
known outcomes: flights can be late or overbooked, connections can be missed, and fog or 
emergency maintenance can cause delays. 



Touring and travelling salesperson problems 

Consider the problem, "Visit every city in Figure 3.3 at least once, starting and ending in 
Bucharest.” This seems very similar to route finding, because the operators still correspond to 
trips between adjacent cities. But for this problem, the state space must record more information. 
In addition to the agent's location, each state must keep track of the set of cities the agent has 
visited. So the initial state would be "In Bucharest; visited {Bucharest},” a typical intermediate 
state would be "In Vaslui; visited {Bucharest, Urziceni.Vaslui} ,” and the goal test would check if 
the agent is in Bucharest and that all 20 cities have been visited. 
salesperson The travelling salesperson problem (TSP) is a famous touring problem in which each city 

must be visited exactly once. The aim is to find the shortest tour. 6 The problem is NP-hard ( Karp, 
1972), but an enormous amount of effort has been expended to improve the capabilities of TSP 
algorithms. In addition to planning trips for travelling salespersons, these algorithms have been 
used for tasks such as planning movements of automatic circuit board drills. 



VLSI layout 

The design of silicon chips is one of the most complex engineering design tasks currently 
undertaken, and we can give only a brief sketch here. A typical VLSI chip can have as many as 
a million gates, and the positioning and connections of every gate are crucial to the successful 
operation of the chip. Computer-aided design tools are used in every phase of the process. 
Two of the most difficult tasks are cell layout and channel routing. These come after the 
components and connections of the circuit have been fixed; the purpose is to lay out the circuit 
on the chip so as to minimize area and connection lengths, thereby maximizing speed. In cell 
layout, the primitive components of the circuit are grouped into cells, each of which performs 
some recognized function. Each cell has a fixed footprint (size and shape) and requires a certain 
number of connections to each of the other cells. The aim is to place the cells on the chip so 
that they do not overlap and so that there is room for the connecting wires to be placed between 
the cells. Channel routing finds a specific route for each wire using the gaps between the cells. 
These search problems are extremely complex, but definitely worth solving. In Chapter 4, we 
will see some algorithms capable of solving them. 



Robot navigation 

Robot navigation is a generalization of the route-finding problem described earlier. Rather than 
a discrete set of routes, a robot can move in a continuous space with (in principle) an infinite set 
of possible actions and states. For a simple, circular robot moving on a flat surface, the space 

6 Strictly speaking, this is the travelling salesperson optimization problem; the TSP itself asks if a tour exists with cost 
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is essentially two-dimensional. When the robot has arms and legs that must also be controlled, 
the search space becomes many-dimensional. Advanced techniques are required just to make 
the search space finite. We examine some of these methods in Chapter 25. In addition to the 
complexity of the problem, real robots must also deal with errors in their sensor readings and 
motor controls. 



Assembly sequencing 

Automatic assembly of complex objects by a robot was first demonstrated by FREDDY the 
robot (Michie. 1972). Progress since then has been slow but sure, to the point where assembly of 
objects such as electric motors is economically feasible. In assembly problems, the problem is to 
find an order in which to assemble the parts of some object. If the wrong order is chosen, there 
will be no way to add some part later in the sequence without undoing some of the work already 
done. Checking a step in the sequence for feasibility is a complex geometrical search problem 
closely related to robot navigation. Thus, the generation of legal successors is the expensive part 
of assembly sequencing, and the use of informed algorithms to reduce search is essential. 



3.4 SEARCHING FOR SOLUTIONS 



We have seen how to define a problem, and how to recognize a solution. The remaining part — 
finding a solution — is done by a search through the state space. The idea is to maintain and extend 
a set of partial solution sequences. In this section, we show how to generate these sequences and 
how to keep track of them using suitable data structures. 

Generating action sequences 

To solve the route-finding problem from Arad to Bucharest, for example, we start offwithjust the ' 
initial state, Arad. The first step is to test if this is a goal state. Clearly it is not, but it is important . 
to check so that we can solve trick problems like "starting in Arad, get to Arad." Because this is . 
not a goal state, we need to consider some other states. This is done by applying the operators ; 
generating to the current state, thereby generating a new set of states. The process is called expanding the 

EXPANDING state. In this case, we get three new states, "in Sibiu,” "in Timisoara.” and "in Zerind,” because 

there is a direct one-step route from Arad to these three cities. If there were only one possibility, 
we wouldjust take it and continue. But whenever there are multiple possibilities, we must make j 
a choice about which one to consider further. 

This is the essence of search — choosing one option and putting the others aside for later, in 
case the first choice does not lead to a solution. Suppose we choose Zerind. We check to see if it 
is a goal state (it is not), and then expand it to get "in Arad" and "in Oradea.” We can then choose 
any of these two, or go back and choose Sibiu or Timisoara. We continue choosing, testing, and 
expanding until a solution is found, or until there are no more states to be expanded. The choice 
of which state to expand first is determined by the search strategy. 
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It is helpful to think of the search process as building up a search tree that is superimposed 
over the state space. The root of the search tree is a search node corresponding to the initial 
state. The leaf nodes of the tree correspond to states that do not have successors in the tree, 
either because they have not been expanded yet, or because they were expanded, but generated 
the empty set. At each step, the search algorithm chooses one leaf node to expand. Figure 3.8 
shows some of the expansions in the search tree for route finding from Arad to Bucharest. The 
general search algorithm is described informally in Figure 3.9. 

It is important to distinguish between the state space and the search tree. For the route- 
finding problem, there are only 20 states in the state space, one for each city. But there are an 




function GENERAL-SEARCH( problem, strategy ) returns a solution, or failure 
initialize the search tree using the initial state of problem 

loop do 

if there are no candidates for expansion then return failure 
choose a leaf node for expansion according to strategy 
if the node contains a goal state then return the corresponding solution 
else expand the node and add the resulting nodes to the search tree 



Figure 3.9 



informal description of the general search algorithm. 
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infinite number of paths in this state space, so the search tree has an infinite number of nodes. 
For example, in Figure 3.8, the branch Arad-Sibiu-Arad continues Arad-Sibiu-Arad-Sibiu- 
Arad, and so on, indefinitely. Obviously, a good search algorithm avoids following such paths. 
Techniques for doing this are discussed in Section 3.6. 



Data structures for search trees 

There are many ways to represent nodes, but in this chapter, we will assume a node is a data 
structure with five components: 

• the state in the state space to which the node corresponds; 

• the node in the search tree that generated this node (this is called the parent node); 

• the operator that was applied to generate the node; 

• the number of nodes on the path from the root to this node (the depth of the node); 

• the path cost of the path from the initial state to the node. 

The node data type is thus: 



datatype node 

components: STATE, PARENT-NODE, OPERATOR, DEPTH, PATH-COST 



It is important to remember the distinction between nodes and states. A node is a bookkeeping 
data structure used to represent the search tree for a particular problem instance as generated by 
a particular algorithm. A state represents a configuration (or set of configurations) of the world. 
Thus, nodes have depths and parents, whereas states do not. (Furthermore, it is quite possible for 
two different nodes to contain the same state, if that state is generated via two different sequences 
of actions.) The EXPAND function is responsible for calculating each of the components of the 
nodes it generates. 

We also need to represent the collection of nodes that are waiting to be expanded — this 
fringe collection is called the fringe or frontier. The simplest representation would be a set of nodes. 

frontier The search strategy then would be a function that selects the next node to be expanded from 

this set. Although this is conceptually straightforward, it could be computationally expensive, 
because the strategy function might have to look at every element of the set to choose the best 
queue one. Therefore, we will assume that the collection of nodes is a implemented as a queue. The 

operations on a queue are as follows: 

• M AKE-QUEUE(E/eme/i f .r preates a queue with the given elements. 

• Empty?(0kcmc) returns true only if there are no more elements in the queue. 

• Rf.MOVE-Front(Qmcmc) removes the element at the front of the queue and returns it. 

• QUEUING-FN(£/emc«f.v,<2uene inserts a set of elements into the queue. Different varieties 
of the queuing function produce different varieties of the search algorithm. 

With these definitions, we can write a more formal version of the general search algorithm. This 
is shown in Figure 3.10. 
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function GENERAL-SEARCH(proWem, QUEUING-FN) returns a solution, or failure 

nodes *- MAKE-QUEUE(MAKE-NODE(lNITIAL-STATE[problem])) 

loop do 

if nodes is empty then return failure 
node — REMOVE-FRONTtnoder) 

if GOAL-TEST[p/wWm] applied to State {node) succeeds then return node 
nodes <- QUEUlNG-FN(node.f, Exvm\tnode. OPERATORS [problem])) 

end 



Figure 3.10 The general search algorithm. (Note that Queuing-Fn is a variable whose value 
will be a function.) 



3.5 SEARCH STRATEGIES 



COMPLETENESS 
TIME COMPLEXITY 
SPACE COMPLEXITY 
OPTIMALITY 






INFORMED SEARCH 

HEURISTIC search 



The majority of work in the area of search has gone into finding the right search strategy for a 
problem. In our study of the field we will evaluate strategies in terms of four criteria: 

0 Completeness: is the strategy guaranteed to find a solution when there is one? 

<0 Time complexity: how long does it take to find a solution? 

0 Space complexity: how much memory does it need to perform the search? 

O' Optimality: does the strategy find the highest-quality solution when there are several 
different solutions? 7 

This section covers six search strategies that come under the heading of uninformed 
search. The term means that they have no information about the number of steps or the path cost 
from the current state to the goal — all they can do is distinguish a goal state from a nongoal state. 
Uninformed search is also sometimes called blind search. 

Consider again the route-finding problem. From the initial state in Arad, there are three 
actions leading to three new states: Sibiu, Timisoara, and Zerind. An uninformed search has 
no preference among these, but a more clever agent might notice that the goal, Bucharest, is 
southeast of Arad, and that only Sibiu is in that direction, so it is likely to be the best choice. 
Strategies that use such considerations are called informed search strategies or heuristic search 
strategies, and they will be covered in Chapter 4. Not surprisingly, uninformed search is less 
effective than informed search. Uninformed search is still important, however, because there are 
many problems for which there is no additional information to consider. 

The six uninformed search strategies are distinguished by the order in which nodes are 
expanded. It turns out that this difference can matter a great deal, as we shall shortly see. 
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Breadth-first search 

One simple search strategy is a breadth-first search. In this strategy, the root node is expanded 
first, then all the nodes generated by the root node are expanded next, and then their successors, 
and so on. In general, all the nodes at depth d in the search tree are expanded before the nodes at 
depth d + 1 . Breadth-first search can be implemented by calling the GENERAL- SEARCH algorithm 
with a queuing function that puts the newly generated states at the end of the queue, after all the 
previously generated states: 



function BREADTH-FlRST-SEARCH( problem) returns a solution or failure 
return GENERAL-SEARCH( pro Wem,ENQUEUEAT-END) 



Breadth-first search is a very systematic strategy because it considers all the paths of length 1 
first, then all those of length 2, and so on. Figure 3.11 shows the progress of the search on a 
simple binary tree. If there is a solution, breadth-first search is guaranteed to find it, and if there 
are several solutions, breadth-first search will always find the shallowest goal state first. In terms 
of the four criteria, breadth-first search is complete, and it is optimal provided the path cost is a 
nondecreasingfunction of the depth of the node. (This condition is usually satisfied only when 
all operators have the same cost. For the general case, see the next section.) 

So far, the news about breadth-first search has been good. To see why it is not always the 
strategy of choice, we have to consider the amount of time and memory it takes to complete a 
search. To do this, we consider a hypothetical state space where every state can be expanded to 
branching factor yield b new states. We say that the branching factor of these states (and of the search tree) is b. 

The root of the search tree generates b nodes at the first level, each of which generates b more 
nodes, for a total of b 2 at the second level. Each of these generates b more nodes, yielding b 3 
nodes at the third level, and so on. Now suppose that the solution for this problem has a path 
length of d. Then the maximum number of nodes expanded before finding a solution is 
1 +b+b 2 +b 3 + • • • +b d 

This is the maximum number, but the solution could be found at any point on the ofth level. In 
the best case, therefore, the number would be smaller. 

Those who do complexity analysis get nervous (or excited, if they are the sort of people 
who like a challenge) whenever they see an exponential complexity bound like 0(b d ). Figure 3.12 
shows why. It shows the time and memory required for a breadth-first search with branching 
factor b = 10 and for various values of the solution depth d. The space complexity is the same 
as the time complexity, because all the leaf nodes of the tree must be maintained in memory 




Figure 3.11 Breadth-first search trees after 0, 1,2, and 3 node expansions. 
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at the same time. Figure 3.12 assumes that 1000 nodes can be goal-checked and expanded per 
second, and that a node requires 100 bytes of storage. Many puzzle-like problems fit roughly 
within these assumptions (give or take a factor of 100) when run on a modem personal computer 
or workstation. 



Depth 


Nodes 


Time 


Memory 


0 


1 


1 millisecond 


100 bytes 


2 


111 


. 1 seconds 


1 1 kilobytes 


4 


11,111 


11 seconds 


1 megabyte 


6 


10 6 


18 minutes 


1 1 1 megabytes 


8 


10 8 


31 hours 


1 1 gigabytes 


10 


10'° 


128 days 


1 terabyte 


12 


10 12 


35 years 


1 1 1 terabytes 


14 


10 14 


3500 years 


11,111 terabytes 


Figure 3.12 Time and memory requirements for breadth-first search. The figures shown 
assume branching factor b = 10; 1000 nodes/second; 1 00 bytes/node. 
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There are two lessons to be learned from Figure 3.12. First, the memory requirements are 
a bigger problem for breadth-first search than the execution time. Most people have the patience 
to wait 18 minutes for a depth 6 search to complete, assuming they care about the answer, but not 
so many have the 111 megabytes of memory that are required. And although 3 1 hours would not 
be too long to wait for the solution to an important problem of depth 8, very few people indeed 
have access to the 11 gigabytes of memory it would take. Fortunately, there are other search 
strategies that require less memory. 

The second lesson is that the time requirements are still a major factor. If your problem 
has a solution at depth 12, then (given our assumptions) it will take 35 years for an uninformed 
search to find it. Of course, if trends continue then in 10 years, you will be able to buy a computer 
that is 100 times faster for the same price as your current one. Even with that computer, however, 
it will still take 128 days to find a solution at depth 12 — and 35 years for a solution at depth 
14. Moreover, there are no other uninformed search strategies that fare any better. In general, 
exponential complexity search problems cannot be solvedfor any but the smallest instances. 

Uniform cost search 

Breadth-first search finds the shallowest goal state, but this may not always be the least-cost 
solution for a general path cost function. Uniform cost search modifies the breadth-first strategy 
by always expanding the lowest-cost node on the fringe (as measured by the path cost g(n)), 
rather than the lowest-depth node. It is easy to see that breadth-first search is just uniform cost 
search with g(n) = Depth (n). 

When certain conditions are met, the first solution that is found is guaranteed to be the 
cheapest solution, because if there were a cheaper path that was a solution, it would have been 
expanded earlier, and thus would have been found first. A look at the strategy in action will help 
explain. Consider the route-finding problem in Figure 3.13. The problem is to get from S to G, 
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and the cost of each operator is marked. The strategy first expands the initial state, yielding paths 
to A, B, and C. Because the path to A is cheapest, it is expanded next, generating the path SAG, 
which is in fact a solution, though not the optimal one. However, the algorithm does not yet 
recognize this as a solution, because it has cost 1 1 , and thus is buried in the queue below the path 
SB, which has cost 5. It seems a shame to generate a solutionjust to bury it deep in the queue, 
but it is necessary if we want to find the optimal solution rather than just any solution. The next 
step is to expand SB, generating SBG, which is now the cheapest path remaining in the queue, so 
it is goal-checked and returned as the solution. 

Uniform cost search finds the cheapest solution provided a simple requirement is met: the 
cost of a path must never decrease as we go along the path. In other words, we insist that 

g(SucCESSOR(n)) > g(n) 
for every node n. 

The restriction to nondecreasing path cost makes sense if the path cost of a node is taken to 
be the sum of the costs of the operators that make up the path. If every operator has a nonnegative 
cost, then the cost of a path can never decrease as we go along the path, and uniform-cost search 
can find the cheapest path without exploring the whole search tree. But if some operator had a 
negative cost, then nothing but an exhaustive search of all nodes would find the optimal solution, 
because we would never know when a path, no matter how long and expensive, is about to run 
into a step with high negative cost and thus become the best path overall. (See Exercise 3.5.) 
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Depth-first search 

Search IHSf Depth-first search always expands one of the nodes at the deepest level of the tree. Only when 

the search hits a dead end (a nongoal node with no expansion) does the search go back and 
expand nodes at shallower levels. This strategy can be implemented by GENERAL-SEARCH with 
a queuing function that always puts the newly generated states at the front of the queue. Because 
the expanded node was the deepest, its successors will be even deeper and are therefore now the 
deepest. The progress of the search is illustrated in Figure 3.14. 

Depth-first search has very modest memory requirements. As the figure shows, it needs 
to store only a single path from the root to a leaf node, along with the remaining unexpanded 
sibling nodes for each node on the path. For a state space with branching factor b and maximum 
depth m. depth-first search requires storage of only bm nodes, in contrast to the b d that would be 
required by breadth-first search in the case where the shallowest goal is at depth d. Using the 
same assumptions as Figure 3.12, depth-first search would require 12 kilobytes instead of 11 1 
terabytes at depth d = 12, a factor of 10 billion times less space. 

The time complexity for depth-first search is 0(b m ). For problems that have very many 
solutions, depth-first may actually be faster than breadth-first, because it has a good chance of 





78 



Chapter 3. 



Solving Problems by Searching 



I 



finding a solution after exploring only a small portion of the whole space. Breadth-first search 
would still have to look at all the paths of length d - 1 before considering any of length d. 
Depth- first search is still 0(b m ) in the worst case. 

The drawback of depth-first search is that it can get stuck going down the wrong path. 
Many problems have very deep or even infinite search trees, so depth-first search will never be 
able to recover from an unlucky choice at one of the nodes near the top of the tree. The search 
will always continue downward without backing up, even when a shallow solution exists. Thus, 
on these problems depth-first search will either get stuck in an infinite loop and never return a 
solution, or it may eventually find a solution path that is longer than the optimal solution. That 
means depth-first search is neither complete nor optimal. Because of this, depth-first search 
should be avoidedfor search trees with large or infinite maximum depths. 

It is trivial to implement depth-first search with GENERAL-SEARCH: 




It is also common to implement depth-first search with a recursive function that calls itself on 
each of its children in turn. In this case, the queue is stored implicitly in the local state of each 
invocation on the calling stack. 



Depth-limited search 

Depth-limited search avoids the pitfalls of depth-first search by imposing a cutoff on the max- 
imum depth of a path. This cutoff can be implemented with a special depth-limited search 
algorithm, or by using the general search algorithm with operators that keep track of the depth. 
For example, on the map of Romania, there are 20 cities, so we know that if there is a solution, 
then it must be of length 19 at the longest. We can implement the depth cutoff using operators 
of the form "If you are in city A and have travelled a path of less than 19 steps, then generate 
a new state in city B with a path length that is one greater." With this new operator set, we are 
guaranteed to find the solution if it exists, but we are still not guaranteed to find the shortest 
solution first: depth-limited search is complete but not optimal. If we choose a depth limit that 
is too small, then depth-limited search is not even complete. The time and space complexity of 
depth-limited search is similar to depth-first search. It takes Oib'fi ime and 0(bl) space, where / 
is the depth limit. 



Iterative deepening search 

The hard part about depth-limited search is picking a good limit. We picked 19 as an "obvious" 
depth limit for the Romania problem, but in fact if we studied the map carefully, we would 
discover that any city can be reached from any other city in at most 9 steps. This number, known 
as the diameter of the state space, gives us a better depth limit, which leads to a more efficient 
depth-limited search. However, for most problems, we will not know a good depth limit until we 
have solved the problem. 
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iTE epeningSEARCH Iterative deepening search is a strategy that sidesteps the issue of choosing the best depth 

° E limit by trying all possible depth limits: first depth 0, then depth 1, then depth 2, and so on. 

The algorithm is shown in Figure 3.15. In effect, iterative deepening combines the benefits of 
depth-first and breadth-first search. It is optimal and complete, like breadth-first search, but has 
only the modest memory requirements of depth-first search. The order of expansion of states is 
similar to breadth-first, except that some states are expanded multiple times. Figure 3.16 shows 
the first four iterations of ITERATIVE-DEEPENING-SEARCH on a binary search tree. 

Iterative deepening search may seem wasteful, because so many states are expanded 
multiple times. For most problems, however, the overhead of this multiple expansion is actually 
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rather small. Intuitively, the reason is that in an exponential search tree, almost all of the nodes 
are in the bottom level, so it does not matter much that the upper levels are expanded multiple 
times. Recall that the number of expansions in a depth-limited search to depth d with branching 
factor b is 

1 + b + b 2 + ■■■ + b d ~ 2 + b d ~ 1 + b d 
To make this concrete, for b = 10 and d = 5. the number is 
1 + 1 0 + 1 00 + 1 , 000 + 1 0, 000 + 1 00, 000 = 1 1 1 , 1 1 1 
In an iterative deepening search, the nodes on the bottom level are expanded once, those on the 
next to bottom level are expanded twice, and so on, up to the root of the search tree, which is 
expanded d + 1 times. So the total number of expansions in an iterative deepening search is 
(d+ 1)1 +{d)b+ (d-\)b 2 + ■■■ + 3b d ~ 2 +2b d - l + 1 b d 
Again, for b = 10 and d = 5 the number is 

6 + 50 + 400 + 3,000 + 20, 000 + 100, 000 = 123,456 
All together, an iterative deepening search from depth 1 all the way down to depth d expands 
only about 1 1 % more nodes than a single breadth-first or depth-limited search to depth d, when 
b = 10. The higher the branching factor, the lower the overhead of repeatedly expanded states, 
but even when the branching factor is 2, iterative deepening search only takes about twice as long 
as a complete breadth-first search. This means that the time complexity of iterative deepening is 
- still 0(b d ), and the space complexity is 0(bd). In general, iterative deepening is the preferred 

* ' search method when there is a large search space and the depth of the solution is not known. 

Bidirectional search 

The idea behind bidirectional search is to simultaneously search both forward from the initial state j 
and backward from the goal, and stop when the two searches meet in the middle (Figure 3.17). j 
For problems where the branching factor is b in both directions, bidirectional search can make a j 
big difference. If we assume as usual that there is a solution of depth d , then the solution will ; 
be found in 0(2b dl2 )= 0(b d/2 ) steps, because the forward and backward searches each have to 1 
go only half way. To make this concrete: for b = 10 and d = 6, breadth-first search generates 
1,111,111 nodes, whereas bidirectional search succeeds when each direction is at depth 3, at 
which point 2,222 nodes have been generated. This sounds great in theory. Several issues needj 
to be addressed before the algorithm can be implemented. 

• The main question is, what does it mean to search backwards from the goal? We define j 

predecessors the predecessors of a node n to be all those nodes that have n as a successor. Searching j 

backwards means generating predecessors successively starting from the goal node. i 

• When all operators are reversible, the predecessor and successor sets are identical; for 
some problems, however, calculating predecessors can be very difficult. 

• What can be done if there are many possible goal states? If there is an explicit list of goal 
states, such as the two goal states in Figure 3.2, then we can apply a predecessor function 
to the state set just as we apply the successor function in multiple-state search. If we only 
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have a description of the set, it may be possible to figure out the possible descriptions of 
"sets of states that would generate the goal set," but this is a very tricky thing to do. For 
example, what are the states that are the predecessors of the checkmate goal in chess? 

• There must be an efficient way to check each new node to see if it already appears in the 
search tree of the other half of the search. 

• We need to decide what kind of search is going to take place in each half. For example, 
Figure 3.17 shows two breadth- first searches. Is this the best choice? 

The 0{b m ) complexity figure assumes that the process of testing for intersection of the two 
frontiers can be done in constant time (that is, is independent of the number of states). This often 
can be achieved with a hash table. In order for the two searches to meet at all, the nodes of at 
least one of them must all be retained in memory (as with breadth-first search). This means that 
the space complexity of uninformed bidirectional search is 0(b dl2 ). 



Comparing search strategies 

Figure 3.18 compares the six search strategies in terms of the four evaluation criteria set forth in 
Section 3.5. 
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Figure 3.18 Evaluation of search strategies, b 
m is the maximum depth of the search tree; / is 


is the branching factor; d is the depth of solution; 
the depth limit. 
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3.6 AVOIDING REPEATED STATES 



Up to this point, we have all but ignored one of the most important complications to the search 
process: the possibility of wasting time by expanding states that have already been encountered 
and expanded before on some other path. For some problems, this possibility never comes up; 
each state can only be reached one way. The efficient formulation of the 8-queens problem is 
efficient in large part because of this — each state can only be derived through one path. 

For many problems, repeated states are unavoidable. This includes all problems where 
the operators are reversible, such as route-finding problems and the missionaries and cannibals 
problem. The search trees for these problems are infinite, but if we prune some of the repeated 
states, we can cut the search tree down to finite size, generating only the portion of the tree that 
spans the state space graph. Even when the tree is finite, avoiding repeated states can yield an 
exponential reduction in search cost. The classic example is shown in Figure 3.19. The space 
contains only m + 1 states, where m is the maximum depth. Because the tree includes each 
possible path through the space, it has 2 m branches. 

There are three ways to deal with repeated states, in increasing order of effectiveness and 
computational overhead: 

• Do not return to the state you just came from. Have the expand function (or the operator 
set) refuse to generate any successor that is the same state as the node's parent. 

• Do not create paths with cycles in them. Have the expand function (or the operator set) 
refuse to generate any successor of a node that is the same as any of the node's ancestors. 

• Do not generate any state that was ever generated before. This requires every state that is 
generated to be kept in memory, resulting in a space complexity of 0(b d ), potentially. It is 
better to think of this as O(s), where s is the number of states in the entire state space. 

To implement this last option, search algorithms often make use of a hash table that stores all 
the nodes that are generated. This makes checking for repeated states reasonably efficient. The 
trade-off between the cost of storing and checking and the cost of extra search depends on the 
problem; the “loopier” the state space, the more likely it is that checking will pay off. 
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A constraint satisfaction problem (or CSP) is a special kind of problem that satisfies some 
additional structural properties beyond the basic requirements for problems in general. In a CSP, 
the states are defined by the values of a set of variables and the goal test specifies a set of 
constraints that the values must obey. For example, the 8-queens problem can be viewed as a 
CSP in which the variables are the locations of each of the eight queens; the possible values are 
squares on the board; and the constraints state that no two queens can be in the same row, column 
or diagonal. A solution to a CSP specifies values for all the variables such that the constraints 
are satisfied. Cryptarithmetic and VLSI layout can also be described as CSPs (Exercise 3.20). 
Many kinds of design and scheduling problems can be expressed as CSPs, so they form a very 
important subclass. CSPs can be solved by general-purpose search algorithms, but because of 
their special structure, algorithms designed specifically for CSPs generally perform much better. 

Constraints come in several varieties. Unary constraints concern the value of a single vari- 
able. For example, the variables corresponding to the leftmost digit on any row of a cryptarithmetic 
puzzle are constrained not to have the value 0. Binary constraints relate pairs of variables. The 
constraints in the 8-queens problem are all binary constraints. Higher-order constraints involve 
three or more variables — for example, the columns in the cryptarithmetic problem must obey 
an addition constraint and can involve several variables. Finally, constraints can be absolute 
constraints, violation of which rules out a potential solution, or preference constraints that say 
which solutions are preferred. 

Each variable V, in a CSP has a domain D,, which is the set of possible values that the 
variable can take on. The domain can be discrete or continuous. In designing a car, for instance, 
the variables might include component weights (continuous) and component manufacturers (dis- 
crete). A unary constraint specifies the allowable subset of the domain, and a binary constraint 
between two variables specifies the allowable subset of the cross-product of the two domains. In 
discrete CSPs where the domains are finite, constraints can be represented simply by enumerating 
the allowable combinations of values. For example, in the 8-queens problem, let V\ be the row 
that the first queen occupies in the first column, and let Vj be the row occupied by the second 
queen in the second column. The domains of V\ and V7 are {1,2, 3,4,5, 6,7, 8}. The no-attack 
constraint linking V\ and V3 can be represented by a set of pairs of allowable values for V\ and 
Vy. {(1,3), (1,4), (1,5), . . . , (2,4), (2, 5), . . .} and so on. Altogether, the no-attack constraint 
between V\ and V-> rules out 22 of the 64 possible combinations. Using this idea of enumeration, 
any discrete CSP can be reduced to a binary CSP. 

Constraints involving continuous variables cannot be enumerated in this way, and solving 
continuous CSPs involves sophisticated algebra. In this chapter, we will handle only discrete, 
absolute, binary (or unary) constraints. Such constraints are still sufficiently expressive to handle 
a wide variety of problems and to introduce most of the interesting solution methods. 

Eet us first consider how we might apply a general-purpose search algorithm to a CSP. The 
initial state will be the state in which all the variables are unassigned. Operators will assign a 
value to a variable from the set of possible values. The goal test will check if all variables are 
assigned and all constraints satisfied. Notice that the maximum depth of the search tree is fixed 
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at n, the number of variables, and that all solutions are at depth n. We are therefore safe in using 
depth-first search, as there is no danger of going too deep and no arbitrary depth limit is needed. 

In the most naive implementation, any unassigned variable in a given state can be assigned 
a value by an operator, in which case the branching factor would be as high as JA D-, \ . or 
64 in the 8-queens problem. A better approach is to take advantage of the fact that the order 
of variable assignments makes no difference to the final solution. Almost all CSP algorithms 
therefore generate successors by choosing values for only a single variable at each node. For 
example, in the 8-queens problem, one can assign a square for the first queen at level 0, for the 
second queen at level 1, and so on. This results in a search space of size njAl, or 8 8 in the 
8-queens problem. A straightforward depth-first search will examine all of these possibilities. 
Because CSPs include as special cases some well-known NP-complete problems such as 3SAT 
(see Exercise 6.15 on page 182), we cannot expect to do better than exponential complexity in 
the worst case. In most real problems, however, we can take advantage of the problem structure 
to eliminate a large fraction of the search space. The principal source of structure in the problem 
space is that, in CSPs, the goal test is decomposed into a set of constraints on variables rather 
than being a "black box. " 

Depth-first search on a CSP wastes time searching when constraints have already been 
violated. Because of the way that the operators have been defined, an operator can never redeem 
a constraint that has already been violated. For example, suppose that we put the first two 
queens in the top row. Depth-first search will examine all 8 6 possible positions for the remaining 
six queens before discovering that no solution exists in that subtree. Our first improvement is 
therefore to insert a test before the successor generation step to check whether any constraint has 
been violated by the variable assignments made up to this point. The resulting algorithm, called 
backtracking search, then backtracks to try something else. 

Backtracking also has some obvious failings. Suppose that the squares chosen for the first I 
six queens make it impossible to place the eighth queen, because they attack all eight squares in 
the last column. Backtracking will try all possible placings for the seventh queen, even though 
the problem is already rendered unsolvable, given the first six choices. Forward checking 
avoids this problem by looking ahead to detect unsolvability. Each time a variable is instantiated, 
forward checking deletes from the domains of the as-yet-uninstantiated variables all of those 
values that conflict with the variables assigned so far. If any of the domains becomes empty, then 
the search backtracks immediately. Forward checking often runs far faster than backtracking and 
is very simple to implement (see Exercise 3.21). ! 

Forward checking is a special case of arc consistency checking. A state is arc-consistent 
if every variable has a value in its domain that is consistent with each of the constraints on that] 
variable. Arc consistency can be achieved by successive deletion of values that are inconsistent: 
with some constraint. As values are deleted, other values may become inconsistent because they 
relied on the deleted values. Arc consistency therefore exhibits a form of constraint propagation, 
as choices are gradually narrowed down. In some cases, achieving arc consistency is enough to 
solve the problem completely because the domains of all variables are reduced to singletons. Arc 
consistency is often used as a preprocessing step, but can also be used during the search. 

Much better results can often be obtained by careful choice of which variable to instantiate 
and which value to try. We examine such methods in the next chapter. 
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This chapter has introduced methods that an agent can use when it is not clear which immediate 
action is best. In such cases, the agent can consider possible sequences of actions; this process is 
called search. 

• Before an agent can start searching for solutions, it must formulate a goal and then use the 
goal to formulate a problem. 

• A problem consists of four parts: the initial state, a set of operators, a goal test function, 
and a path cost function. The environment of the problem is represented by a state space. 
A path through the state space from the initial state to a goal state is a solution. 

• In real life most problems are ill-defined; but with some analysis, many problems can fit 
into the state space model. 

• A single general search algorithm can be used to solve any problem; specific variants of 
the algorithm embody different strategies. 

• Search algorithms are judged on the basis of completeness, optimality, time complexity, 
and space complexity. Complexity depends on b, the branching factor in the state space, 
and d , the depth of the shallowest solution. 

• Breadth-first search expands the shallowest node in the search tree first. It is complete, 
optimal for unit-cost operators, and has time and space complexity of 0(b d ). The space 
complexity makes it impractical in most cases. 

• Uniform-cost search expands the least-cost leaf node first. It is complete, and unlike 
breadth-first search is optimal even when operators have differing costs. Its space and time 
complexity are the same as for breadth-first search. 

• Depth-first search expands the deepest node in the search tree first. It is neither complete 
nor optimal, and has time complexity of 0(b m ) and space complexity of O(bm), where m is 
the maximum depth. In search trees of large or infinite depth, the time complexity makes 
this impractical. 

• Depth-limited search places a limit on how deep a depth-first search can go. If the limit 
happens to be equal to the depth of shallowest goal state, then time and space complexity 
are minimized. 

• Iterative deepening search calls depth-limited search with increasing limits until a goal is 
found. It is complete and optimal, and has time complexity of 0(b d ) and space complexity 
of 0(bd). 

• Bidirectional search can enormously reduce time complexity, but is not always applicable. 
Its memory requirements may be impractical. 
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BIBLIOGRAPHICAL AND HISTORICAL NOTES 

Most of the state-space search problems analyzed in this chapter have a long history in the 
literature, and are far less trivial than they may seem. The missionaries and cannibals problem 
was analyzed in detail from an artificial intelligence perspective by Amarel (1968), although 
Amarel's treatment was by no means the first; it had been considered earlier in AI by Simon 
and Newell (1961 ), and elsewhere in computer science and operations research by Bellman and 
Dreyfus (1962). Studies such as these led to the establishment of search algorithms as perhaps the 
primary tools in the armory of early AI researchers, and the establishment of problem solving as 
the canonical AI task. (Of course, one might well claim that the latter resulted from the former.) 

Amarel's treatment of the missionaries and cannibals problem is particularly noteworthy 
because it is a classic example of formal analysis of a problem stated informally in natural 
language. Amarel gives careful attention to abstracting from the informal problem statement 
precisely those features that are necessary or useful in solving the problem, and selecting a formal 
problem representation that represents only those features. A more recent treatment of problem 
representation and abstraction, including AI programs that themselves perform these tasks (in 
part), is to be found in Knoblock (1990). 

The 8-queens problem was first published anonymously in the German chess magazine 
Schach in 1 848; it was later attributed to one Max Bezzel. It was republished in 1 850, and at that 
time drew the attention of the eminent mathematician Carl Friedrich Gauss, who attempted to 
enumerate all possible solutions. Even Gauss was able to find only 72 of the 92 possible solutions 
offhand, which gives some indication of the difficulty of this apparently simple problem. (Nauck, 
who had republished the puzzle, published all 92 solutions later in 1850.) Netto (1901 (generalized 
the problem to “n-queens" (on annxn chessboard). 

The 8-puzzle initially appeared as the more complex 4x4 version, called the 15-puzzle. It 
was invented by the famous American game designer Sam Loyd ( 1959) in the 1870s and quickly 
achieved immense popularity in the United States, comparable to the more recent sensation caused 
by the introduction of Rubik's Cube. It also quickly attracted the attention of mathematicians 
(Johnson and Story, 1879;Tait, 1880). The popular reception of the puzzle was so enthusiastic 
that the Johnson and Story article was accompanied by a note in which the editors of the American 
Journal of Mathematics felt it necessary to state that "The ‘15’ puzzle for the last few weeks 
has been prominently before the American public, and may safely be said to have engaged the 
attention of nine out of ten persons of both sexes and all ages and conditions of the community. 
But this would not have weighed with the editors to induce them to insert articles upon such a 
subject in the American Journal of Mathematics, but for the fact that . . .” (there follows a brief 
summary of the reasons for the mathematical interest of the 15-puzzle). The 8-puzzle has often 
been used in AI research in place of the 15-puzzle because the search space is smaller and thus 
more easily subjected to exhaustive analysis or experimentation. An exhaustive analysis was 
carried out with computer aid by P. D. A. Schofield (1967). Although very time-consuming, 
this analysis allowed other, faster search methods to be compared against theoretical perfection 
for the quality of the solutions found. An in-depth analysis of the 8-puzzle, using heuristic 
search methods of the kind described in Chapter 4, was carried out by Doran and Michie (1966). 
The 15-puzzle, like the 8-queens problem, has been generalized to the n x n case. Ratner and 





Section 3.8. 



Summary 



87 



Warmuth ( 1 986) showed that finding the shortest solution in the generalized nxn version belongs 
to the class of NP-complete problems. 

"Uninformed" search algorithms for finding shortest paths that rely on current path cost 
alone, rather than an estimate of the distance to the goal, are a central topic of classical computer 
science, applied mathematics, and a related field known as operations research. Uniform-cost 
search as a way of finding shortest paths was invented by Dijkstra (1959). A survey of early 
work in uninformed search methods for shortest paths can be found in Dreyfus (1969); Deo 
and Pang (1982) give a more recent survey. For the variant of the uninformed shortest-paths 
problem that asks for shortest paths between all pairs of nodes in a graph, the techniques of 
dynamic programming and memoization can be used. For a problem to be solved by these 
techniques, it must be capable of being divided repeatedly into subproblems in such a way 
that identical subproblems arise again and again. Then dynamic programming or memoization 
involves systematically recording the solutions to subproblems in a table so that they can be 
looked up when needed and do not have to be recomputed repeatedly during the process of 
solving the problem. An efficient dynamic programming algorithm for the all-pairs shortest-paths 
problem was found by Bob Floyd (1962a; 1962b), and improved upon by Karger et al. (1993). 
Bidirectional search was introduced by Pohl (1969; 1 971); it is often used with heuristic guidance 
techniques of the kind discussed in Chapter 4. Iterative deepening was first used by Slate and 
Atkin (1977) in the CHESS 4.5 game-playing program. 

The textbooks by Nilsson (1971; 1980) are good general sources of information about 
classical search algorithms, although they are now somewhat dated. A comprehensive, and much 
more up-to-date, survey can be found in (Korf, 1988). 



EXERCISES 

3.1 Explain why problem formulation must follow goal formulation. 

3.2 Consider the accessible, two-location vacuum world under Murphy's Law. Show that for 
each initial state, there is a sequence of actions that is guaranteed to reach a goal state. 

3.3 Give the initial state, goal test, operators, and path cost function for each of the following. 
There are several possible formulations for each problem, with varying levels of detail. The 
main thing is that your formulations should be precise and "hang together" so that they could be 
implemented. 

a. You want to find the telephone number of Mr. Jimwill Zollicoffer, who lives in Alameda, 
given a stack of directories alphabetically ordered by city. 

b. As for part (a), but you have forgotten Jimwill's last name. 

c. You are lost in the Amazon jungle, and have to reach the sea. There is a stream nearby. 

d. You have to color a complex planar map using only four colors, with no two adjacent 
regions to have the same color. 
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e. A monkey is in a room with a crate, with bananas suspendedjust out ofreach on the ceiling. 
He would like to get the bananas. 

f. You are lost in a small country town, and must find a drug store before your hay fever 
becomes intolerable. There are no maps, and the natives are all locked indoors. 

3.4 Implement the missionaries and cannibals problem and use breadth-first search to find the 
shortest solution. Is it a good idea to check for repeated states? Draw a diagram of the complete 
state space to help you decide. 

35 On page 76, we said that we would not consider problems with negative path costs. In this 
exercise, we explore this in more depth. 

a. Suppose that a negative lower bound c is placed on the cost of any given step — that is, 
negative costs are allowed, but the cost of a step cannot be less than c. Does this allow 
uniform-cost search to avoid searching the whole tree? 

b. Suppose that there is a set of operators that form a loop, so that executing the set in some 
order results in no net change to the state. If all of these operators have negative cost, what 
does this imply about the optimal behavior for an agent in such an environment? 

c. One can easily imagine operators with high negative cost, even in domains such as route- i 
finding. For example, some stretches of road might have such beautiful scenery as to far j 
outweigh the normal costs in terms of time and fuel. Explain, in precise terms, why humans ; 
do not drive round scenic loops indefinitely, and explain how to define the state space and I 
operators for route-finding so that artificial agents can also avoid looping. 

d. Can you think of a real domain in which step costs are such as to cause looping? 

3.6 The GENERAL- SEARCH algorithm consists of three steps: goal test, generate, and ordering 
function, in that order. It seems a shame to generate a node that is in fact a solution, but to fail tol 
recognize it because the ordering function fails to place it first. 

a. Write a version of GENERAL-SEARCH that tests each node as soon as it is generated and 
stops immediately if it has found a goal. 

b. Show how the GENERAL-SEARCH algorithm can be used unchanged to do this by giving it 

the proper ordering function. \ 

3.7 The formulation of problem, solution, and search algorithm given in this chapter explicitly i 
mentions the path to a goal state. This is because the path is important in many problems. Fori 
other problems, the path is irrelevant, and only the goal state matters. Consider the problem; 
"Find the square root of 123454321." A search through the space of numbers may pass through] 
many states, but the only one that matters is the goal state, the number 1 1 1 1 1 . Of course, from a 1 
theoretical point of view, it is easy to run the general search algorithm and then ignore all of the f 
path except the goal state. But as a programmer, you may realize an efficiency gain by coding a j 
version of the search algorithm that does not keep track of paths. Consider a version of problem I 
solving where there are no paths and only the states matter. Write definitions of problem and 
solution, and the general search algorithm. Which of the problems in Section 3.3 would best use 
this algorithm, and which should use the version that keeps track of paths? 
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3.8 Given a pathless search algorithm such as the one called for in Exercise 3.7, explain how 
you can modify the operators to keep track of the paths as part of the information in a state. Show 
the operators needed to solve the route-finding and touring problems. 

3.9 Describe a search space in which iterative deepening search performs much worse than 
depth-first search. 

3.10 Figure 3.17 shows a schematic view of bidirectional search. Why do you think we chose 
to show trees growing outward from the start and goal states, rather than two search trees growing 
horizontally toward each other? 

3.11 Write down the algorithm for bidirectional search, in pseudo-code or in a programming 
language. Assume that each search will be a breadth-first search, and that the forward and 
backward searches take turns expanding a node at a time. Be careful to avoid checking each node 
in the forward search against each node in the backward search! 

3.12 Give the time complexity of bidirectional search when the test for connecting the two 
searches is done by comparing a newly generated state in the forward direction against all the 
states generated in the backward direction, one at a time. 

3.13 We said that at least one direction of a bidirectional search must be a breadth-first search. 
What would be a good choice for the other direction? Why? 

3.14 Consider the following operator for the 8-queens problem: place a queen in the column 
with the fewest unattacked squares, in such a way that it does not attack any other queens. How 
many nodes does this expand before it finds a solution? (You may wish to have a program 
calculate this for you.) 

3.15 The chain problem (Figure 3.20) consists of various lengths of chain that must be recon- 
figured into new arrangements. Operators can open one link and close one link. In the standard 
form of the problem, the initial state contains four chains, each with three links. The goal state 
consists of a single chain of 12 links in a circle. Set this up as a formal search problem and find 
the shortest solution. 
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3.16 Tests of human intelligence often contain sequence prediction problems. The aim in 
such problems is to predict the next member of a sequence of integers, assuming that the number 
in position n of the sequence is generated using some sequence function s(n), where the first 
element of the sequence corresponds to n = 0. For example, the function s(n) - 2" generates the 
sequence [1,2, 4,8, 16, ...]. 

In this exercise, you will design a problem-solving system capable of solving such pre- 
diction problems. The system will search the space of possible functions until it finds one that 
matches the observed sequence. The space of sequence functions that we will consider consists 
of all possible expressions built from the elements 1 and n, and the functions +, x, — , /, and 
exponentiation. For example, the function 2" becomes (1 + 1)" in this language. It will be useful 
to think of function expressions as binary trees, with operators at the internal nodes and 1 's and 
n s at the leaves. 

a. First, write the goal test function. Its argument will be a candidate sequence function ,v. It 
will contain the observed sequence of numbers as local state. 

b. Now write the successor function. Given a function expression ,v, it should generate all 
expressions one step more complex than s. This can be done by replacing any leaf of the 
expression with a two-leaf binary tree. 

c. Which of the algorithms discussed in this chapter would be suitable for this problem? 
Implement it and use it to find sequence expressions for the sequences [1,2, 3, 4, 5], 
[1,2,4,8,16, . . .], and [0.5, 2,4.5, 8], 

d. If level d ofthe search space contains all expressions of complexity d + 1 , where complexity 
is measured by the number of leaf nodes (e.g., n + (l x n ) has complexity 3), prove by 
induction that there are roughly 20 d (d+ 1)! expressions at level d. 

e. Comment on the suitability of uninformed search algorithms for solving this problem. Can 
you suggest other approaches? 



The full vacuum world from the exercises in Chapter 2 can be viewed as a search problem 
sense we have defined, provided we assume that the initial state is completely known. 



a. Define the initial state, operators, goal test function, and path cost function. 

b. Which of the algorithms defined in this chapter would be appropriate for this problem? 

c. Apply one of them to compute an optimal sequence of actions for a 3 x 3 world with dirt in 
the center and home squares. 

d. Construct a search agent for the vacuum world, and evaluate its performance in a set of 
3x3 worlds with probability 0.2 of dirt in each square. Include the search cost as well as 
path cost in the performance measure, using a reasonable exchange rate. 

e. Compare the performance of your search agent with the performance of the agents con- 
structed for the exercises in Chapter 2. What happens if you include computation time in 
the performance measure, at various "exchange rates" with respect to the cost of taking a 
step in the environment? 

f. Consider what would happen if the world was enlarged to nxn. How does the performance 
of the search agent vary with nt Of the reflex agents? 
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l: , 3.18 The search agents we have discussed make use of a complete model of the world to 

construct a solution that they then execute. Modify the depth- first search algorithm with repeated 
state checking so that an agent can use it to explore an arbitrary vacuum world even without a 
model of the locations of walls and dirt. It should not get stuck even with loops or dead ends. 
You may also wish to have your agent construct an environment description of the type used by 
the standard search algorithms. 

3.19 In discussing the cryptarithmetic problem, we proposed that an operator should assign 
a value to whichever letter has the least remaining possible values. Is this rule guaranteed to 
produce the smallest possible search space? Why (not)? 

3.20 Define each of the following as constraint satisfaction problems: 

a. The cryptarithmetic problem. 

b. The channel-routing problem in VLSI layout. 

map-coloring c. The map-coloring problem. In map-coloring, the aim is to color countries on a map using 

a given set of colors, such that no two adjacent countries are the same color. 
floor-planning d. The rectilinear floor-planning problem, which involves finding nonoverlapping places in 

a large rectangle for a number of smaller rectangles. 

3.21 Implement a constraint satisfaction system as follows: 

a. Define a datatype forCSPs with finite, discrete domains. You will need to find a way to 
represent domains and constraints. 

b. Implement operators that assign values to variables, where the variables are assigned in a 
fixed order at each level of the tree. 

c. Implement a goal test that checks a complete state for satisfaction of all the constraints. 

d. Implement backtracking by mo dify ing DEPTH-FIRST-SEARCH . 

e. Add forward checking to your backtracking algorithm. 

f. Run the three algorithms on some sample problems and compare their performance. 
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INFORMED SEARCH 
METHODS 



In which we see how information about the state space can prevent algorithms from 
blundering about in the dark. 



Chapter 3 showed that uninformed search strategies can find solutions to problems by systemati- 
cally generating new states and testing them against the goal. Unfortunately, these strategies are 
incredibly inefficient in most cases. This chapter shows how an informed search strategy— one 
that uses problem-specific knowledge — can find solutions more efficiently. It also shows how 
optimization problems can be solved. 



4.1 BEST-FIRST SEARCH 



In Chapter 3, we found several ways to apply knowledge to the process of formulating a problem 
in terms of states and operators. Once we are given a well-defined problem, however, our options 
are more limited. If we plan to use the GENERAL-SEARCH algorithm from Chapter 3, then 
the only place where knowledge can be applied is in the queuing function, which determines 
the node to expand next. Usually, the knowledge to make this determination is provided by an 
function N evaluation function that returns a number purporting to describe the desirability (or lack thereof) 

of expanding the node. When the nodes are ordered so that the one with the best evaluation is 
best-first search expanded first, the resulting strategy is called best-first search. It can be implemented directly 
with GENERAL-SEARCH, as shown in Figure 4. 1 . 

The name "best-first search" is a venerable but inaccurate one. After all, if we could really 
expand the best node first, it would not be a search at all; it would be a straight march to the goal. 
All we can do is choose the node that appears to be best according to the evaluation function. 
If the evaluation function is omniscient, then this will indeed be the best node; in reality, the 
evaluation function will sometimes be off, and can lead the search astray. Nevertheless, we will 
stick with the name "best-first search," because “seemingly-best-first search" is a little awkward. 

Just as there is a whole family of GENERAL-SEARCH algorithms with different ordering 
functions, there is also a whole family of BEST-FIRST-SEARCH algorithms with different evaluation 
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function Best-First-Search( problem, EVAL-FN) returns a solution sequence 
inputs: problem, a problem 

Eval-Fn, an evaluation function 

Queueing-Fn^a function that orders nodes by EVAL-FN 
return GENERAL-SEARCH( problem, Queueing-Fn) 



Figure 4.1 An implementation of best-first search using the general search algorithm. 



functions. Because they aim to find low-cost solutions, these algorithms typically use some 
estimated measure of the cost of the solution and try to minimize it. We have already seen one 
such measure: the use of the path cost g to decide which path to extend. This measure, however, 
does not direct search toward the goal. In order tofocus the search, the measure must incorporate 
some estimate of the cost of the pathfrom a state to the closest goal state. We look at two basic 
approaches. The first tries to expand the node closest to the goal. The second tries to expand the 
node on the least-cost solution path. 




STRAIGHT-LIN 

DISTANCE 



Minimize estimated cost to reach a goal: Greedy search 

One of the simplest best-first search strategies is to minimize the estimated cost to reach the goal. 
That is, the node whose state is judged to be closest to the goal state is always expanded first. 
For most problems, the cost of reaching the goal from a particular state can be estimated but 
cannot be determined exactly. A function that calculates such cost estimates is called a heuristic 
function, and is usually denoted by the letter h: 

h(n) = estimated cost of the cheapest path from the state at node n to a goal state. 

A best-first search that uses h to select the next node to expand is called greedy search, for 
reasons that will become clear. Given a heuristic function h, the code for greedy search is just 
the following: 



function Greedy-Search( problem) returns a solution or failure 
return Best-First-Search( problem, h) 



Formally speaking, h can be any function at all. We will require only that h(n) = 0 if n is a goal. 

To get an idea of what a heuristic function looks like, we need to choose a particular 
problem, because heuristic functions are problem-specific. Let us return to the route-finding 
problem from Arad to Bucharest. The map for that problem is repeated in Figure 4.2. 

A good heuristic function for route-finding problems like this is the straight-line distance 
to the goal. That is, 

hsix>(n) = straight-line distance between n and the goal location. 
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HISTORY OF "HEURISTIC" 




By now the space aliens had mastered my own language, but they still made 
simple mistakes like using “hermeneutic” when they meant ''heuristic.'' 

— a Louisiana factory worker in Woody Alien's The UFO Menace 
The word "heuristic" is derived from the Greek verb heuriskein, meaning "to find" 
or "to discover." Archimedes is said to have run naked down the street shouting 
"H eureka" (I have found it) after discovering the principle of flotation in his bath. 
Later generations converted this to Eureka. 

The technical meaning of "heuristic" has undergone several changes in the history 
of AI . In 1 957, George Polya wrote an influential book called How to Solve It that used 
"heuristic" to refer to the study of methods for discovering and inventing problem- 
solving techniques, particularly for the problem of coming up with mathematical 
proofs. Such methods had often been deemed not amenable to explication. 

Some people use heuristic as the opposite of algorithmic. For example, Newell, 
Shaw, and Simon stated in 1963, "A process that may solve a given problem, but offers 
no guarantees of doing so, is called a heuristic for that problem.” But note that there 
is nothing random or nondeterministic about a heuristic search algorithm: it proceeds 
by algorithmic steps toward its result. In some cases, there is no guarantee how long 
the search will take, and in some cases, the quality of the solution is not guaranteed 
either. Nonetheless, it is important to distinguish between “nonalgorithmic” and "not 
precisely characterizable.” 

Heuristic techniques dominated early applications of artificial intelligence. The 
first "expert systems" laboratory, started by Ed Feigenbaum, Brace Buchanan, and 
Joshua Lederberg at Stanford University, was called the Heuristic Programming 
Project (HPP). Heuristics were viewed as "rules of thumb” that domain experts could 
use to generate good solutions without exhaustive search. Heuristics were initially 
incorporated directly into the structure of programs, but this proved too inflexible 
when a large number of heuristics were needed. Gradually, systems were designed 
that could accept heuristic information expressed as "rules," and rule-based systems 
were born. 

Currently, heuristic is most often used as an adjective, referring to any technique 
that improves the average-case performance on a problem-solving task, but does 
not necessarily improve the worst-case performance. In the specific area of search 
algorithms, it refers to a function that provides an estimate of solution cost. 

A good article on heuristics (and one on hermeneutics!) appears in the Encyclo- 
pedia ofAI (Shapiro, 1992). 
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Notice that we can only calculate the values of hsLD if we know the map coordinates of the cities 
in Romania. Furthermore, hsLD is only useful because a road from A to B usually tends to head 
in more or less the right direction. This is the sort of extra information that allows heuristics to 
help in reducing search cost. 

Figure 4.3 shows the progress of a greedy search to find a path from Arad to Bucharest. 
With the straight-line-distance heuristic, the first node to be expanded from Arad will be Sibiu, 
because it is closer to Bucharest than either Zerind or Timisoara. The next node to be expanded 
will be Fagaras, because it is closest. Fagaras in turn generates Bucharest, which is the goal. For 
this particular problem, the heuristic leads to minimal search cost: it finds a solution without ever 
expanding a node that is not on the solution path. However, it is not perfectly optimal: the path 
it found via Sibiu and Fagaras to Bucharest is 32 miles longer than the path through Rimnicu 
Vilcea and Pitesti. This path was not found because Fagaras is closer to Bucharest in straight-line 
distance than Rimnicu Vilcea, so it was expanded first. The strategy prefers to take the biggest 
bite possible out of the remaining cost to reach the goal, without worrying about whether this 
will be best in the long run — hence the name "greedy search." Although greed is considered 
one of the seven deadly sins, it turns out that greedy algorithms often perform quite well. They 
tend to find solutions quickly, although as shown in this example, they do not always find the 
optimal solutions: that would take a more careful analysis of the long-term options, not just the 
immediate best choice. 

Greedy search is susceptible to false starts. Consider the problem of getting from Iasi to 
Fagaras. The heuristic suggests that Neamt be expanded first, but it is a dead end. The solution is 
to go first to Vaslui — a step that is actually farther from the goal according to the heuristic — and 
then to continue to Urziceni, Bucharest, and Fagaras. Hence, in this case, the heuristic causes 
unnecessary nodes to be expanded. Furthermore, if we are not careful to detect repeated states, 
the solution will never be found — the search will oscillate between Neamt and Iasi. 
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Greedy search resembles depth-first search in the way it prefers to follow a single path all 
the way to the goal, but will back up when it hits a dead end. It suffers from the same defects 
as depth-first search — it is not optimal, and it is incomplete because it can start down an infinite 
path and never return to try other possibilities. The worst-case time complexity for greedy search 
is 0(b m ), where m is the maximum depth of the search space. Because greedy search retains 
all nodes in memory, its space complexity is the same as its time complexity. With a good 
heuristic function, the space and time complexity can be reduced substantially. The amount of 
the reduction depends on the particular problem and quality of the h function. 



Minimizing the total path cost: A* search 

Greedy search minimizes the estimated cost to the goal, h{n), and thereby cuts the search cost 
considerably. Unfortunately, it is neither optimal nor complete. Uniform-cost search, on the 
other hand, minimizes the cost of the path so far, g(n)\ it is optimal and complete, but can be 
very inefficient. It would be nice if we could combine these two strategies to get the advantages 
of both. Fortunately, we can do exactly that, combining the two evaluation functions simply by 
summing them: 



/(«) = g(n) + h(n). 
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Since g(ri) gives the path cost from the start node to node n. and h{n) is the estimated cost of the 
cheapest path from n to the goal, we have 

f(n) = estimated cost of the cheapest solution through n 
Thus, if we are trying to find the cheapest solution, a reasonable thing to try first is the node with 
the lowest value of /.The pleasant thing about this strategy is that it is more than just reasonable. 
We can actually prove that it is complete and optimal, given a simple restriction on the h function. 

The restriction is to choose an h function that never overestimates the cost to reach the 
hmstic e goal. Such an h is called an admissible heuristic. Admissible heuristics are by nature optimistic, 

because they think the cost of solving the problem is less than it actually is. This optimism 
• transfers to the/ function as well: If his admissible, f(n) never overestimates the actual cost of 

lil? the best solution through n. Best-first search using/ as the evaluation function and an admissible 

a* search h function is known as A* search 




Perhaps the most obvious example of an admissible heuristic is the straight-line distance 
hsLD that we used in getting to Bucharest. Straight-line distance is admissible because the shortest 
path between any two points is a straight line. In Figure 4.4, we show the first few steps of an A* 
search for Bucharest using the hsw heuristic. Notice that the A* search prefers to expand from 
Rimnicu Vilcea rather than from Fagaras. Even though Fagaras is closer to Bucharest, the path 
taken to get to Fagaras is not as efficient in getting close to Bucharest as the path taken to get to 
Rimnicu. The reader may wish to continue this example to see what happens next. 



The behavior of A* search 

Before we prove the completeness and optimality of A*, it will be useful to present an intuitive 
picture of how it works. A picture is not a substitute for a proof, but it is often easier to remember 
and can be used to generate the proof on demand. First, a preliminary observation: if you examine 
the search trees in Figure 4.4, you will notice an interesting phenomenon. Along any path from 
the root, the /-cost never decreases. This is no accident. It holds true for almost all admissible 
monotonicity heuristics. A heuristic for which it holds is said to exhibit monotonicity . 1 

If the heuristic is one of those odd ones that is not monotonic, it turns out we can make a 
minor correction that restores monotonicity. Let us consider two nodes n and n' , where n is the 
parent ofn'. Now suppose, for example, that g(n) = 3 and h(n) = 4. Then fin) = g(n)+h(n)- 7 — 
that is, we know that the true cost of a solution path through n is at least 7. Suppose also that 
gin') = 4 and h(n')- 2, so that/(n') = 6. Clearly, this is an example of a nonmonotonic heuristic. 
Fortunately, from the fact that any path through n' is also a path through n, we can see that the 
value of 6 is meaningless, because we already know the true cost is at least 7. Thus, we should 

1 It can be proved (Pearl, 1 984 ) that a heuristic is monotonic if and only if it obeys the triangle inequality. The triangle 
inequality says that two sides of a triangle cannot add up to less than the third side (see Exercise 4.7). Of course, 
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check, each time we generate a new node, to see if its/-cost is less than its parent’s/-cost; if it 
is, we use the parent’s/-cost instead: 

/(«') = max(f(n)g(ri) + h(n')). 

In this way, we ignore the misleading values that may occur with a nonmonotonic heuristic. This 
pathmax equation is called the pathmax equation. If we use it, then /will always be nondecreasing along 

any path from the root, provided h is admissible. 

The purpose of making this observation is to legitimize a certain picture of what A* does. 
CONTOURS If /never decreases along any path out from the root, we can conceptually draw contours in the 

state space. Figure 4.5 shows an example. Inside the contour labelled 400, all nodes have/(«) 
less than or equal to 400, and so on. Then, because A* expands the leaf node of lowest /, we can 
see that an A* search fans out from the start node, adding nodes in concentric bands of increasing 
/-cost. 

With uniform-cost search (A* search using h = 0), the bands will be "circular" around the 
start state. With more accurate heuristics, the bands will stretch toward the goal state and become 
more narrowly focused around the optimal path. If we define f* to be the cost of the optimal 
solution path, then we can say the following: 

• A* expands all nodes with /(n) </*. 

• A* may then expand some of the nodes right on the "goal contour," for which f(n) =/*, 
before selecting a goal node. 

Intuitively, it is obvious that the first solution found must be the optimal one, because nodes in 
all subsequent contours will have higher /-cost, and thus higher g-cost (because all goal states 
have h(n) = 0). Intuitively, it is also obvious that A* search is complete. As we add bands of 
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increasing /, we must eventually reach a band where / is equal to the cost of the path to a goal 
state. We will turn these intuitions into proofs in the next subsection. 

One final observation is that among optimal algorithms of this type — algorithms that extend 
search paths from the root — A* is optimally efficient for any given heuristic function. That is, 
no other optimal algorithm is guaranteed to expand fewer nodes than A*. We can explain this 
as follows: any algorithm that does not expand all nodes in the contours between the root and 
the goal contour runs the risk of missing the optimal solution. A long and detailed proof of this 
result appears in Dechter and Pearl (1985). 



Proof of the optimality of A* 

Let G be an optimal goal state, with path cost/*. Let G 2 be a suboptimal goal state, that is, a 
goal state with path cost g(G2) >/*. The situation we imagine is that A* has selected G2 from 
the queue. Because Gi is a goal state, this would terminate the search with a suboptimal solution 
(Figure 4.6). We will show that this is not possible. 

Consider a node n that is currently a leaf node on an optimal path to G (there must be some 
such node, unless the path has been completely expanded — in which case the algorithm would 
have returned G). Because h is admissible, we must have 

r >/(«)• 

Furthermore, if n is not chosen for expansion over Gj. we must have 

f(n)>nG 2 }, 

Combining these two together, we get 

r >/(g 2 ). 



100 



Chapter 4. Informed Search Methods 




But because G2is a goal state, we have h(G 2 ) = 0; hence /(G2) = g(G 2). Thus, we have proved, 
from our assumptions, that 

/* > g(G 2 ). 

This contradicts the assumption that G 2 is suboptimal, so it must be the case that A* never selects 
a suboptimal goal for expansion. Hence, because it only returns a solution after selecting it for 
expansion, A* is an optimal algorithm. 



Proof of the completeness of A* 

We said before that because A* expands nodes in order of increasing/, it must eventually expand 
to reach a goal state. This is true, of course, unless there are infinitely many nodes with/(«) </* . 
The only way there could be an infinite number of nodes is either (a) there is a node with an 
infinite branching factor, or (b) there is a path with a finite path cost but an infinite number of 
nodes along it. 2 

Thus, the correct statement is that A* is complete on locally finite graphs (graphs with a 
finite branching factor) provided there is some positive constant 6 such that every operator costs 
at least 8. 



Complexity of A* 

That A* search is complete, optimal, and optimally efficient among all such algorithms is rather 
satisfying. Unfortunately, it does not mean that A* is the answer to all our searching needs. The 
catch is that, for most problems, the number of nodes within the goal contour search space is still 
exponential in the length of the solution. Although the proof of the result is beyond the scope of 
this book, it has been shown that exponential growth will occur unless the error in the heuristic 

2 Zeno’s paradox, which purports to show that a rock thrown at a tree will never reach it, provides an example that 
violates condition (b). The paradox is created by imagining that the trajectory is divided into a series of phases, each of 
which covers half the remaining distance to the tree; this yields an infinite sequence of steps with a finite total cost. 
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function grows no faster than the logarithm of the actual path cost. In mathematical notation, the 
condition for subexponential growth is that 

\h(ri)~ h*{n)\ < 0(\ogh*(n)), 

where h* ( n)is the true cost of getting from n to the goal. For almost all heuristics in practical use, 
the error is at least proportional to the path cost, and the resulting exponential growth eventually 
overtakes any computer. Of course, the use of a good heuristic still provides enormous savings 
compared to an uninformed search. In the next section, we will look at the question of designing 
good heuristics. 

Computation time is not, however, A*'s main drawback. Because it keeps all generated 
nodes in memory, A* usually runs out of space long before it runs out of time. Recently developed 
algorithms have overcome the space problem without sacrificing optimality or completeness. 
These are discussed in Section 4.3. 



4.2 HEURISTIC FUNCTIONS 



So far we have seen just one example of a heuristic: straight-line distance for route-finding 
problems. In this section, we will look at heuristics for the 8-puzzle. This will shed light on the 
nature of heuristics in general. 

The 8-puzzle was one of the earliest heuristic search problems. As mentioned in Section 3.3, 
the object of the puzzle is to slide the tiles horizontally or vertically into the empty space until 
the initial configuration matches the goal configuration (Figure 4.7). 




Figure 4.7 A typical instance of the 8-puzzle. 



The 8-puzzle is just the right level of difficulty to be interesting. A typical solution is about 
20 steps, although this of course varies depending on the initial state. The branching factor is 
about 3 (when the empty tile is in the middle, there are four possible moves; when it is in a comer 
there are two; and when it is along an edge there are three). This means that an exhaustive search 
to depth 20 would look at about 3 20 = 3.5 x 10 9 states. By keeping track of repeated states, we 
could cut this down drastically, because there are only 9! = 362, 880 different arrangements of 
9 squares. This is still a large number of states, so the next order of business is to find a good 
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heuristic function. If we want to find the shortest solutions, we need a heuristic function that 
never overestimates the number of steps to the goal. Here are two candidates: 

• h\ = the number of tiles that are in the wrong position. For Figure 4.7, none of the 8 tiles 
is in the goal position, so the start state would have h\ = 8. h\ is an admissible heuristic, 
because it is clear that any tile that is out of place must be moved at least once. 

• h 2 = the sum of the distances of the tiles from their goal positions. Because tiles cannot 
move along diagonals, the distance we will count is the sum of the horizontal and vertical 
distances. This is sometimes called the city block distance or Manhattan distance. h 2 is 
also admissible, because any move can only move one tile one step closer to the goal. The 
8 tiles in the start state give a Manhattan distance of 

h 2 = 2 + 3 + 2+1+2+2+1+2=15 



The effect of heuristic accuracy on performance 

One way to characterize the quality of a heuristic is the effective branching factor b*. If the' 
total number of nodes expanded by A* for a particular problem is N, and the solution depth is 
d, then b* is the branching factor that a uniform tree of depth d would have to have in order to 
contain N nodes. Thus, 



N= 1 +b*+(b*f+ ■ 



*■ 



For example, if A* finds a solution at depth 5 using 52 nodes, then the effective branching factor is 
1 .91 . Usually, the effective branching factor exhibited by a given heuristic is fairly constant over 
a large range of problem instances, and therefore experimental measurements of b" on a small 
set of problems can provide a good guide to the heuristic's overall usefulness. A well-designed- 
heuristic would have a value of b* close to 1, allowing fairly large problems to be solved. To 
test the heuristic functions h\ and h 2 , we randomly generated 100 problems each with solution 
lengths 2, 4, ... , 20, and solved them using A* search with h\ and h 2 , as well as with uninformed 
iterative deepening search. Figure 4.8 gives the average number of nodes expanded by each 
strategy, and the effective branching factor. The results show that h 2 is better than h\, and that i 
uninformed search is much worse. 

One might ask if h 2 is always better than h\ . The answer is yes. It is easy to see from the j 
definitions of the two heuristics that for any node n, h 2 (n) > h\(n). We say that h 2 dominates hi- 
Domination translates directly into efficiency: A* using h 2 will expand fewer nodes, on average , \ 
than A* using h\ . We can show this using the following simple argument. Recall the observation 
on page 98 that every node with/(n) </* will be expanded. This is the same as saying that every 
node with h{n) </* — g(n) will be expanded. But because h 2 is at least as big as h\ for all nodes, 
every node that is expanded by A* search with h 2 will also be expanded with hi, and h\ may also I 
cause other nodes to be expanded as well. Hence, it is always better to use a heuristic functior 
with higher values, as long as it does not overestimate. 
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Figure 4.8 Comparison of the search costs and effective branching factors for the 
ITERATIVE-DEEPENING-SEARCH and A* algorithms with h\ hi- Data are averaged over 100 
instances of the 8-puzzle, for various solution lengths. 



Inventing heuristic functions 

We have seen that both h\ and h 2 are fairly good heuristics for the 8-puzzle, and that h 2 is better. 
But we do not know how to invent a heuristic function. How might one have come up with h^l 
Is it possible for a computer to mechanically invent such a heuristic? 

h\ and h 2 are estimates to the remaining path length for the 8-puzzle, but they can also 
be considered to be perfectly accurate path lengths for simplified versions of the puzzle. If 
the rules of the puzzle were changed so that a tile could move anywhere, instead of just to the 
adjacent empty square, then h\ would accurately give the number of steps to the shortest solution. 
Similarly, if a tile could move one square in any direction, even onto an occupied square, then h 2 
would give the exact number of steps in the shortest solution. A problem with less restrictions on 
RELAXED PROBLEM the operators is called a relaxed problem. It is often the case that the cost of an exact solution 
l to a relaxed problem is a good heuristic for the original problem. 

1 If a problem definition is written down in a formal language, it is possible to construct 

relaxed problems automatically. 3 For example, if the 8-puzzle operators are described as 
A tile can move from square A to square B if A is adjacent to B and B is blank, 
we can generate three relaxed problems by removing one or more of the conditions: 

(a) A tile can move from square A to square B if A is adjacent to B. 

(b) A tile can move from square A to square B if B is blank. 

(c) A tile can move from square A to square B. 

Recently, a program called ABSOLVER was written that can generate heuristics automatically from 
problem definitions, using the "relaxed problem" method and various other techniques (Prieditis, 
1993). ABSOLVER generated a new heuristic for the 8-puzzle better than any existing heuristic, 
and found the first useful heuristic for the famous Rubik's cube puzzle. 

3 tn Chapters 7 and 11 , we will describe formal languages suitable for this task. For now, we will use English. 
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One problem with generating new heuristic functions is that one often fails to get one 
"clearly best" heuristic. If a collection of admissible heuristics h\...h m is available for a 
problem, and none of them dominates any of the others, which should we choose? As it turns 
out, we need not make a choice. We can have the best of all worlds, by defining 

h(n) = max(h ] (n),...,h m (n)). . 

This composite heuristic uses whichever function is most accurate on the node in question. 
Because the component heuristics are admissible, h is also admissible. Furthermore, h dominates 
all of the individual heuristics from which it is composed. 

Another way to invent a good heuristic is to use statistical information. This can be 
gathered by running a search over a number of training problems, such as the 100 randomly 
chosen 8-puzzle configurations, and gathering statistics. For example, we might find that when 
/i 2 («) = 14, it turns out that 90% of the time the real distance to the goal is 18. Then when faced 
with the "real" problem, we can use 18 as the value whenever /? 2 (n) reports 14. Of course, if we 
use probabilistic information like this, we are giving up on the guarantee of admissibility, but we 
are likely to expand fewer nodes on average. 

FEATURES Often it is possible to pick out features of a state that contribute to its heuristic evaluation 

function, even if it is hard to say exactly what the contribution should be. For example, the goal 
in chess is to checkmate the opponent, and relevant features include the number of pieces of each 
kind belonging to each side, the number of pieces that are attacked by opponent pieces, and so 
on. Usually, the evaluation function is assumed to be a linear combination of the feature values. 
Even if we have no idea how important each feature is, or even if a feature is good or bad, it 
is still possible to use a learning algorithm to acquire reasonable coefficients for each feature, 
as demonstrated in Chapter 18. In chess, for example, a program could learn that one's own 
queen should have a large positive coefficient, whereas an opponent's pawn should have a small 
negative coefficient. 

Another factor that we have not considered so far is the search cost of actually running the 
heuristic function on a node. We have been assuming that the cost of computing the heuristic 
function is about the same as the cost of expanding a node, so that minimizing the number of 
nodes expanded is a good thing. But if the heuristic function is so complex that computing its 
value for one node takes as long as expanding hundreds of nodes, then we need to reconsider. 
After all, it is easy to have a heuristic that is perfectly accurate — if we allow the heuristic to do, 
say, a full breadth-first search "on the sly." That would minimize the number of nodes expanded 
by the real search, but it would not minimize the overall search cost. A good heuristic function 
must be efficient as well as accurate. 



Heuristics for constraint satisfaction problems 

In Section 3.7, we examined a class ofproblems called constraint satisfaction problems (CSPs). 
A constraint satisfaction problem consists of a set of variables that can take on values from a given 
domain, together with a set of constraints that specify properties of the solution. Section 3.7 
examined uninformed search methods for CSPs, mostly variants of depth-first search. Here, 
we extend the analysis by considering heuristics for selecting a variable to instantiate and for 
choosing a value for the variable. 
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To illustrate the basic idea, we will use the map-coloring problem shown in Figure 4.9. 
(The idea of map coloring is to avoid coloring adjacent countries with the same color.) Suppose 
that we can use at most three colors (red, green, and blue), and that we have chosen green for 
country A and red for country B . Intuitively, it seems obvious that we should color E next, because 
the only possible color for E is blue. All the other countries have a choice of colors, and we 
might make the wrong choice and have to backtrack. In fact, once we have colored E blue, then 
we are forced to color C red and F green. After that, coloring D either blue or red results in a 
solution. In other words, we have solved the problem with no search at all. 




This intuitive idea is called the most-constrained- variable heuristic. It is used with 
forward checking (see Section 3.7), which keeps track of which values are still allowed for each 
variable, given the choices made so far. At each point in the search, the variable with th efewest 
possible values is chosen to have a value assigned. In this way, the branching factor in the search 
tends to be minimized. For example, when this heuristic is used in solving n-queens problems, 
the feasible problem size is increased from around 30 for forward checking to approximately 
100. The most-constraining- variable heuristic is similarly effective. It attempts to reduce the 
branching factor on future choices by assigning a value to the variable that is involved in the 
largest number of constraints on other unassigned variables. 

Once a variable has been selected, we still need to choose a value for it. Suppose that 
we decide to assign a value to country C after A and B. One's intuition is that red is a better 
choice than blue, because it leaves more freedom for future choices. This intuition is the 
least-constraining- value heuristic — choose a value that rules out the smallest number of values 
in variables connected to the current variable by constraints. When applied to the «-queens 
problem, it allows problems up to n=1000 to be solved. 




106 



Chapter 4. Informed Search Methods 



4.3 MEMORY BOUNDED SEARCH 



Despite all the clever search algorithms that have been invented, the fact remains that some prob- 
lems are intrinsically difficult, by the nature of the problem. When we run up against these expo- 
nentially complex problems, something has to give. Figure 3.12 shows that the first thing to give 
is usually the available memory. In this section, we investigate two algorithms that are designed 
to conserve memory. The first, IDA*, is a logical extension of ITERATIVE-DEEPENING-SEARCH 
to use heuristic information. The second, SMA*, is similar to A*, but restricts the queue size to 
fit into the available memory. 



IDA* 



Iterative deepening A* search (IDA*) 

In Chapter 3, we showed that iterative deepening is a useful technique for reducing memory 
requirements. We can try the same trick again, turning A* search into iterative deepening A*, or 
IDA* (see Figure 4.10). In this algorithm, each iteration is a depth-first search, just as in regular 
iterative deepening. The depth-first search is modified to use an /-cost limit rather than a depth 
limit. Thus, each iteration expands all nodes inside the contour for the current /-cost, peeping 
over the contour to find out where the next contour lies. (See the DFS-Contour function in 
Figure 4.10.) Once the search inside a given contour has been completed, a new iteration is 
started using a new /-cost for the next contour. 

IDA* is complete and optimal with the same caveats as A* search, but because it is depth- 
first, it only requires space proportional to the longest path that it explores. If b is the smallest 
operator cost and/* the optimal solution cost, then in the worst case, IDA* will require bf*IS 
nodes of storage. In most cases, bd is a good estimate of the storage requirements. 

The time complexity of IDA* depends strongly on the number of different values that the 
heuristic function can take on. The Manhattan distance heuristic used in the 8-puzzle takes on 
one of a small number of integer values. Typically,/ only increases two or three times along any 
solution path. Thus, IDA* only goes through two or three iterations, and its efficiency is similar 
to that of A* — in fact, the last iteration of IDA* usually expands roughly the same number of 
nodes as A*. Furthermore, because IDA* does not need to insert and delete nodes on a priority 
queue, its overhead per node can be much less than that for A*. Optimal solutions for many 
practical problems were first found by IDA*, which for several years was the only optimal, 
memory-bounded, heuristic algorithm. 

Unfortunately, IDA* has difficulty in more complex domains. In the travelling salesperson 
problem, for example, the heuristic value is different for every state. This means that each contour 
only includes one more state than the previous contour. If A* expands A nodes, IDA* will have 
to go through A iterations and will therefore expand 1 +2 + ••■ + N = <9(/V 2 )nodes. Now if A is 
too large for the computer's memory, then A 2 is almost certainly too long to wait! 

One way around this is to increase the/-cost limit by a fixed amount e on each iteration, so 
that the total number of iterations is proportional to 1/e. This can reduce the search cost, at the 
expense of returning solutions that can be worse than optimal by at most f . Such an algorithm is 
called f -admissible. 
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function ID A*(profo/em)retums a solution sequence 
inputs: problem, aproblem 
static: f-limit, the current/- COST limit 
mot, anode 

root *- MAKE-NODE(lNlTIAL-STATE|/?raii/e'm]) 
f-limit <— f- CoST(roof) 
loop do 

solution, f-limit — DFS-CONTOUR (root f-limit) 
if solution is non-null then return solution 
it f-limit = oo then return failure; end 



function DFS-Contour (node, f-limit) returns a solution sequence and a new /- COST limit 
inputs: node, a node 

f-limit, the current/- COST limit 
static: next-f the/- COST limit for the next contour, initially oc 

iff - CoSTf node] > f-limit then return null,/- COST [node] 
if Go AL-TEST|/?roWem ](State[ node | ) then return node, f-limit 
for each node s in SucCESSORS(node) do 
solution, new-f — DFS -CONTOUR! .s. /-/«?// 
if solution is non-null then return solution, f-limit 
next-f <— MiN(next-f new-f); end 
return null, next-f 



Figure 4.10 The IDA* (Iterative Deepening A*) search algorithm. 



SMA* search 

IDA*'s difficulties in certain problem spaces can be traced to using too little memory. Between 
iterations, it retains only a single number, the current /-cost limit. Because it cannot remember 
its history, IDA* is doomed to repeat it. This is doubly true in state spaces that are graphs rather 
than trees (see Section 3.6). IDA* can be modified to check the current path for repeated states, 
but is unable to avoid repeated states generated by alternative paths. 
sma* In this section, we describe the SMA* (Simplified Memory-Bounded A*) algorithm, which 

can make use of all available memory to carry out the search. Using more memory can only 
improve search efficiency — onecould always ignore the additional space, but usually it is better to 
remember a node than to have to regenerate it when needed. SMA* has the following properties: 

• It will utilize whatever memory is made available to it. 

• It avoids repeated states as far as its memory allows. 

• It is complete if the available memory is sufficient to store the shallowest solution path. 

• It is optimal if enough memory is available to store the shallowest optimal solution path. 

Otherwise, it returns the best solution that can be reached with the available memory. 

• When enough memory is available for the entire search tree, the search is optimally efficient. 
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The one unresolved question is whether SMA* is always optimally efficient among all algorithms 
given the same heuristic information and the same memory allocation. 

The design of SMA* is simple, at least in overview. When it needs to generate a successor 
but has no memory left, it will need to make space on the queue. To do this, it drops a node from 
the queue. Nodes that are dropped from the queue in this way are called forgotten nodes. It 
prefers to drop nodes that are unpromising — that is, nodes with high/-cost. To avoid reexploring 
subtrees that it has dropped from memory, it retains in the ancestor nodes information about the 
quality of the best path in the forgotten subtree. In this way, it only regenerates the subtree when 
all other paths have been shown to look worse than the path it has forgotten. Another way of 
saying this is that if all the descendants of a node n are forgotten, then we will not know which 
way to go from n, but we will still have an idea of how worthwhile it is to go anywhere from n. 

SMA* is best explained by an example, which is illustrated in Figure 4.1 1. The top of the 
figure shows the search space. Each node is labelled with g + h —/values, and the goal nodes (D, 
F, I, J) are shown in squares. The aim is to find the lowest-cost goal node with enough memory 
for only three nodes. The stages of the search are shown in order, left to right, with each stage 
numbered according to the explanation that follows. Each node is labelled with its current /-cost, 
which is continuously maintained to reflect the least /-cost of any of its descendants. 4 Values in 
parentheses show the value of the best forgotten descendant. The algorithm proceeds as follows: 

1 . At each stage, one successor is added to the deepest lowest-/-cost node that has some 
successors not currently in the tree. The left child B is added to the root A. 

2. Now /(A) is still 12, so we add the right child G (/' = 13). Now that we have seen all the 
children of A, we can update its /-cost to the minimum of its children, that is, 13. The 
memory is now full. 

3. G is now designated for expansion, but we must first drop a node to make room. We drop 
the shallowest highest-/-cost leaf, that is, B. When we have done this, we note that A's best 
forgotten descendant has/ = 15, as shown in parentheses. We then add H, with /(//) =18. 
Unfortunately, H is not a goal node, but the path to H uses up all the available memory. 
Hence, there is no way to find a solution through H, so we set f(H) = oo. 

4. G is expanded again. We drop H, and add I, with f (I) = 24. Now we have seen both 
successors of G, with values of oo and 24, so /(G) becomes 24. /(A) becomes 15, the 
minimum of 15 (forgotten successor value) and 24. Notice that I is a goal node, but it 
might not be the best solution because A's/-cost is only 15. 

5. A is once again the most promising node, so B is generated for the second time. We have 
found that the path through G was not so great after all. 

6. C, the first successor of B, is a nongoal node at the maximum depth, so f(C) = oo. 

7. To look at the second successor, D, we first drop C. Then f(D) = 20, and this value is 
inherited by B and A. 

8. Now the deepest, lowest-/-cost node is D. D is therefore selected, and because it is a goal 
node, the search terminates. 



4 Values computed in this way are called backed-up values. Because/(n) is supposed to be an estimate of the least-cost 
solution path through n, and a solution path through n is bound to go through one of n’s descendants, backing up the least 
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In this case, there is enough memory for the shallowest optimal solution path. If J had had a cost 
of 19 instead of 24, however, SMA* still would not have been able to find it because the solution 
path contains four nodes. In this case, SMA* would have returned D, which would be the best 
reachable solution. It is a simple matter to have the algorithm signal that the solution found may 
not be optimal. 

A rough sketch of SMA* is shown in Figure 4.12. In the actual program, some gymnastics 
are necessary to deal with the fact that nodes sometimes end up with some successors in memory 
and some forgotten. When we need to check for repeated nodes, things get even more complicated. 
SMA* is the most complicated search algorithm we have seen yet. 



function S MA*(problem returns a solution sequence 
inputs: problem, a problem 
static: Queue, a queue of nodes ordered by /-cost 

Queue — MAKE-QUEUE({MAKE-NODE(lNlTlAL-STATE[prob/em])}) 

loop do 

if Queue is empty then return failure 
n — deepest least-f-cost node in Queue 
if GoAL-TEST(n) then return success 
S — Next-Successor( n ) 
if i is not a goal and is at maximum depth then 
f(.s) - oc 
else 

f(s) <- MAX(f(n), g(i)+h(i)) 
if all of n’ s successors have been generated then 

update n’s/-cost and those of its ancestors if necessary 
if SUCCESSORS(n) all in memory then remove n from Queue 
if memory is full then 

delete shallowest, highest-f-cost node in Queue 
remove it from its parent's successor list 
insert its parent on Queue if necessary 
insert 5 on Queue 

end 



Figure 4.12 Sketch of the SMA* algorithm. Note that numerous details have been omitted in 

the interests of clarity. 



Given a reasonable amount of memory, SMA* can solve significantly more difficult prob- 
lems than A* without incurring significant overhead in terms of extra nodes generated. It performs 
well on problems with highly connected state spaces and real-valued heuristics, on which IDA* 
has difficulty. On very hard problems, however, it will often be the case that SMA* is forced 
to continually switch back and forth between a set of candidate solution paths. Then the extra 
time required for repeated regeneration of the same nodes means that problems that would be 
practically solvable by A*, given unlimited memory, become intractable for SMA*. That is to 
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say, memory limitations can make a problem intractable from the point of view of computation 
time. Although there is no theory to explain the trade-off between time and memory, it seems 
that this is an inescapable problem. The only way out is to drop the optimality requirement. 



A A ITERATIVE IMPROVEMENT ALGORITHMS 




HILL-CLIMBING 

gradient descent 

SIMULATED 

ANNEALING 



We saw in Chapter 3 that several well-known problems (for example, 8-queens and VLSI layout) 
have the property that the state description itself contains all the informationneeded for a solution. 
The path by which the solution is reached is irrelevant. In such cases, iterative improvement 
algorithms often provide the most practical approach. For example, we start with all 8 queens on 
the board, or all wires routed through particular channels. Then, we might move queens around 
trying to reduce the number of attacks; or we might move a wire from one channel to another 
to reduce congestion. The general idea is to start with a complete configuration and to make 
modifications to improve its quality. 

The best way to understand iterative improvement algorithms is to consider all the states 
laid out on the surface of a landscape. The height of any point on the landscape corresponds to 
the evaluation function of the state at that point (Figure 4.13). The idea of iterative improvement 
is to move around the landscape trying to find the highest peaks, which are the optimal solutions. 
Iterative improvement algorithms usually keep track of only the current state, and do not look 
ahead beyond the immediate neighbors of that state. This resembles trying to find the top of 
Mount Everest in a thick fog while suffering from amnesia. Nonetheless, sometimes iterative 
improvement is the method of choice for hard, practical problems. We will see several applications 
in later chapters, particularly to neural network learning in Chapter 1 9. 

Iterative improvement algorithms divide into two major classes. Hill-climbing (or, alter- 
natively, gradient descent if we view the evaluation function as a cost rather than a quality) 
algorithms always try to make changes that improve the current state. Simulated annealing 
algorithms can sometimes make changes that make things worse, at least temporarily. 



Hill-climbing search 

The hill-climbing search algorithm is shown in Figure 4.14. It is simply a loop that continually 
moves in the direction of increasing value. The algorithm does not maintain a search tree, so the 
node data structure need only record the state and its evaluation, which we denote by VALUE. 
One important refinement is that when there is more than one best successor to choose from, the 
algorithm can select among them at random. This simple policy has three well-known drawbacks: 

0 Local maxima: a local maximum, as opposed to a global maximum, is a peak that is lower 
than the highest peak in the state space. Once on a local maximum, the algorithm will halt 
even though the solution may be far from satisfactory. 

<0> Plateaux: a plateau is an area of the state space where the evaluation function is essentially 
flat. The search will conduct a random walk. 
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Figure 4.13 Iterative improvement algorithms try to find peaks on a surface of states where 
height is defined by the evaluation function. 



function Hill-Climbing (problem) returns a solution state 
inputs: problem, a problem 
static: current, a node 
next, anode 

current <- MAKE-NODH(lNITIAL-STATE|proW<?m] ) 

loop do 

next <— a highest-valued successor of current 
if VALUEfnext] < VALUEfcurrent] then return current 
current «— next 

end 



Figure 4.14 The hill-climbing search algorithm. 



O’ Ridges: a ridge may have steeply sloping sides, so that the search reaches the top of the 
ridge with ease, but the top may slope only very gently toward a peak. Unless there happen 
to be operators that move directly along the top of the ridge, the search may oscillate from 
side to side, making little progress. 

In each case, the algorithm reaches a point at which no progress is being made. If this happens, an 
™ D o1mbFng ART obvious thing to do is start again from a different starting point. Random-restart hill-climbing 
does just this: it conducts a series of hill-climbing searches from randomly generated initial 
states, running each until it halts or makes no discernible progress. It saves the best result found 
so far from any of the searches. It can use a fixed number of iterations, or can continue until the 
best saved result has not been improved for a certain number of iterations. 
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Clearly, if enough iterations are allowed, random-restart hill-climbing will eventually find 
the optimal solution. The success of hill-climbing depends very much on the shape of the state- 
space "surface": if there are only a few local maxima, random-restart hill-climbing will find a 
good solution very quickly. A realistic problems has surface that looks more like a porcupine. 
If the problem is NP-complete. then in ah likelihood we cannot do better than exponential time. 
It follows that there must be an exponential number of local maxima to get stuck on. Usually, 
however, a reasonably good solution can be found after a small number of iterations. 

Simulated annealing 

Instead of starting again randomly when stuck on a local maximum, we could allow the search 
to take some downhill steps to escape the local maximum. This is roughly the idea of simulated 
annealing (Figure 4.15). The innermost loop of simulated annealing is quite similar to hill- 
climbing. Instead of picking the best move, however, it picks a random move. If the move 
actually improves the situation, it is always executed. Otherwise, the algorithm makes the move 
with some probability less than 1 . The probability decreases exponentially with the "badness" of 
the move — the amount AE by which the evaluation is worsened. 

A second parameter T is also used to determine the probability. At higher values of T, "bad" 
moves are more likely to be allowed. As T tends to zero, they become more and more unlikely, 
until the algorithm behaves more or less like hill-climbing. The schedule input determines the 
value of T as a function of how many cycles already have been completed. 

The reader by now may have guessed that the name "simulated annealing" and the parameter 
names AE and T were chosen for a good reason. The algorithm was developed from an explicit 
analogy with annealing — the process of gradually cooling a liquid until it freezes. The VALUE 
function corresponds to the total energy of the atoms in the material, and T corresponds to the 



function SmVLAJED-ANNEALWG(problem,schedule) returns a solution state 
inputs: problem, a problem 

schedule, a mapping from time to "temperature" 
static: current, a node 
next, a node 

T, a "temperature" controlling the probability of downward steps 

current*— MAKE-NoDE(lNlTlAL-STATE[prob/em]) 
for t <— 1 to oo do 
T — schedule^] 
if T=0then return current 
next <— a randomly selected successor of current 
AE*— VALUEfnerf] - Value [current] 
if AE > 0 then current — next 
else current <— next only with probability f M/I 



Figure 4.15 The simulated annealing search algorithm. 
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temperature. The schedule determines the rate at which the temperature is lowered. Individual 
moves in the state space correspond to random fluctuations due to thermal noise. One can prove 
that if the temperature is lowered sufficiently slowly, the material will attain a lowest-energy 
(perfectly ordered) configuration. This corresponds to the statement that if schedule lowers T 
slowly enough, the algorithm willjfind a global optimum. 

Simulated annealing was first used extensively to solve VLSI layout problems in the 
early 1980s. Since then, it has been applied widely to factory scheduling and other large-scale 
optimization tasks. In Exercise 4.12, you are asked to compare its performance to that of 
random-restart hill-climbing on the n-queens puzzle. 

Applications in constraint satisfaction problems 

Constraint satisfaction problems (CSPs) can be solved by iterative improvement methods by 
first assigning values to all the variables, and then applying modification operators to move the 
configuration toward a solution. Modification operators simply assign a different value to a 
variable. For example, in the 8-queens problem, an initial state has all eight queens on the board, 
and an operator moves a queen from one square to another. 
heuristic repair Algorithms that solve CSPs in this fashion are often called heuristic repair methods, 

because they repair inconsistencies in the current configuration. In choosing a new value for a 
variable, the most obvious heuristic is to select the value that results in the minimum number of 
min-conflicts conflicts with other variables — the min-conflicts heuristic. This is illustrated in Figure 4.16 for 
an 8-queens problem, which it solves in two steps. 

Min-conflicts is surprisingly effective for many CSPs, and is able to solve the million- 
queens problem in an average of less than 50 steps. It has also been used to schedule observations 
for the Hubble space telescope, reducing the time taken to schedule a week of observations from 
three weeks (!) to around ten minutes. Min-conflicts is closely related to the GSAT algorithm 
described on page 182, which solves problems in propositional logic. 




Figure 4.16 A two-step solution for an 8-queens problem using min-conflicts. At each stage, a 
queen is chosen for reassignment in its column. The number of conflicts (in this case, the number 
of attacking queens) is shown in each square. The algorithm moves the queen to the min-conflict 
square, breaking ties randomly. 
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45_ SUMMARY 

This chapter has examined the application of heuristics to reduce search costs. We have looked 
at number of algorithms that use heuristics, and found that optimality comes at a stiff price in 
terms of search cost, even with good heuristics. 

• Best-first search is just GENERAL-SEARCH where the minimum-cost nodes (according to 
some measure) are expanded first. 

• If we minimize the estimated cost to reach the goal, h(n), we get greedy search. The 
search time is usually decreased compared to an uninformed algorithm, but the algorithm 
is neither optimal nor complete. 

• Minimizing/(«) = g(n) + h{n) combines the the advantages of uniform-cost search and 
greedy search. If we handle repeated states and guarantee that h(n) never overestimates, 
we get A* search. 

• A* is complete, optimal, and optimally efficient among all optimal search algorithms. Its 
space complexity is still prohibitive. 

• The time complexity of heuristic algorithms depends on the quality of the heuristic function. 
Good heuristics can sometimes be constructed by examining the problem definition or by 
generalizing from experience with the problem class. 

• We can reduce the space requirement of A* with memory-bounded algorithms such as 
IDA* (iterative deepening A*) and SMA* (simplified memory-bounded A*). 

• Iterative improvement algorithms keep only a single state in memory, but can get stuck 
on local maxima. Simulated annealing provides a way to escape local maxima, and is 
complete and optimal given a long enough cooling schedule. 

• For constraint satisfaction problems, variable and value ordering heuristics can provide 
huge performance gains. Current algorithms often^solve very large problems very quickly. 



BIBLIOGRAPHICAL AND HISTORICAL NOTES 

The same paper that introduced the phrase "heuristic search" (Newell and Ernst, 1965) also 
introduced the concept of an evaluation function, understood as an estimate of the distance to 
the goal, to guide search; this concept was also proposed in the same year by Lin (1965). Doran 
and Michie (1966) did extensive experimental studies of heuristic search as applied to a number 
of problems, especially the 8-puzzle and the 15-puzzle. Although Doran and Michie carried out 
theoretical analyses of path length and "penetrance” (the ratio of path length to the total number 
of nodes examined so far) in heuristic search, they appear to have used their heuristic functions as 
the sole guiding element in the search, ignoring the information provided by current path length 
that is used by uniform-cost search and by A*. 

The A* algorithm, incorporating the current path length into heuristic search, was developed 
by Hart, Nilsson, and Raphael (1968). Certain subtle technical errors in the original presentation 
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of A* were corrected in a later paper (Hart et al., 1972). An excellent summary of early work in 
search is provided by Nilsson (1971). 

The original A* paper introduced a property of heuristics called "consistency." The 
monotonicity property of heuristics was introduced by Pohl (1977) as a simpler replacement for 
the consistency property. Pearl (19.84) showed that consistency and monotonicity were equivalent 
properties. The pathmax equation was first used in A* search by Mero (1984). 

Pohl ( 1970; 1977) pioneered the study of the relationship between the error in A*'s heuristic 
function and the time complexity of A*. The proof that A* runs in linear time if the error in the 
heuristic function is bounded by a constant can be found in Pohl (1977) and in Gaschnig (1979). 
Pearl (1984) strengthened this result to allow a logarithmic growth in the error. The "effective 
branching factor" measure of the efficiency of heuristic search was proposed by Nilsson (1971). 

A* and other state-space search algorithms are closely related to the branch-and-bound 
techniques that are widely used in operations research. An early survey of branch-and-bound 
techniques is given by Lawler and Wood (1966). The seminal paper by Held and Karp (1970) 
considers the use of the minimum-spanning- tree heuristic (see Exercise 4.11) for the travelling 
salesperson problem, showing how such admissible heuristics can be derived by examining 
relaxed problems. Generation of effective new heuristics by problem relaxation was successfully 
implemented by Prieditis (1993), building on earlier work with Jack Mostow (Mostow and 
Prieditis, 1989). The probabilistic interpretation of heuristics was investigated in depth by 
Hansson and Mayer (1989). 

The relationships between state-space search and branch-and-bound have been investigated 
indepthbyDanaNau, Laveen Kanal, and Vipin Kumar (Kumar and Kanal, 1983;Naue?a/., 1984; 
Kumar et al., 1988). Martelli and Montanari (1978) demonstrate a connection between dynamic 
programming (see Chapter 17) and certain types of state-space search. Kumar and Kanal (1988) 
attempt a "grand unification" of heuristic search, dynamic programming, and branch-and-bound 
techniques under the name of CDP — the "composite decision process." More material along 
these lines is found in Kumar (1991 ). 

There are a large number of minor and major variations on the A* algorithm. Pohl (1973) 
proposed the use of dynamic weighting , which uses a weighted sum /„,(«) = w g g(n)+ wi,h(n ) of 
the current path length and the heuristic function as an evaluation function, rather than the simple 
sum /'tn) = g(n) + h(n) used in A*, and dynamically adjusts the weights w g and Wh according 
to certain criteria as the search progresses. Dynamic weighting usually cannot guarantee that 
optimal solutions will be found, as A* can, but under certain circumstances dynamic weighting 
can find solutions much more efficiently than A*. 

The most-constrained-variable heuristic was introduced by Bitnerand Reingold (1975), and 
further investigated by Purdom (1983). Empirical results on the n-queens problem were obtained 
by Stone and Stone (1986). Brelaz (1979) used the most-constraining-variable heuristic as a tie- 
breaker after applying the most-constrained-variable heuristic. The resulting algorithm, despite 
its simplicity, is still the best method fork-coloring arbitrary graphs. The least-constraining-value 
heuristic was developed and analyzed in Haralickand Elliot (1980). The min-conflicts heuristic 
was first proposed by Gu (1989), and was developed independently by Steve Minton (Minton et 
al., 1992). Minton explains the remarkable performance of min-conflicts by modelling the search 
process as a random walk that is biased to move toward solutions. The effectiveness of algorithms 
such as min-conflicts and the related GSAT algorithm (see Exercise 6.15) in solving randomly 
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generated problems almost "instantaneously," despite the NP-completeness of the associated 
problem classes, has prompted an intensive investigation. It turns out that almost all randomly 
generated problems are either trivially easy or have no solutions. Only if the parameters of the 
problem generator are set in a certain narrow range, within which roughly half of the problems 
are solvable, do we find "hard" problem instances (Kirkpatrick and Selman, 1994). 

Because computers in the late 1950s and early 1960s had at most a few thousand words 
of main memory, memory-bounded heuristic search was an early research topic. Doran and 
Michie's (1966) Graph Traverser, one of the earliest search programs, commits to an operator 
after searching best-first up to the memory limit. As with other "staged search" algorithms, 
optimality cannot be ensured because until the best path has been found the optimality of the 
first step remains in doubt. IDA* was the first widely used optimal, memory-bounded, heuristic 
search algorithm, and a large number of variants have been developed. The first reasonably public 
paper dealing specifically with IDA* was Korfs (1985b), although Korf credits the initial idea 
to a personal communication from Judea Pearl and also mentions Berliner and Goetsch's (1984) 
technical report describing their implementation of IDA* concurrently with Korfs own work. 
A more comprehensive exposition of IDA* can be found in Korf (1985a). A thorough analysis 
of the efficiency of IDA*, and its difficulties with real-valued heuristics, appears in Patrick 
et al. (1992). The SMA* algorithm described in the text was based on an earlier algorithm 
called MA* (Chakrabarti et al, 1989), and first appeared in Russell (1992). The latter paper 
also introduced the "contour" representation of search spaces. Kaindl and Khorsand (1994) 
apply SMA* to produce a bidirectional search algorithm that exhibits significant performance 
improvements over previous algorithms. 

Three other memory-bounded algorithms deserve mention. RBFS (Korf, 1993) and 
IE (Russell, 1992) are two very similar algorithms that use linear space and a simple recur- 
sive formulation, like IDA*, but retain information from pruned branches to improve efficiency. 
Particularly in tree-structured search spaces with discrete-valued heuristics, they appear to be 
competitive with SMA* because of their reduced overhead. RBFS is also able to carry out a 
tabu search best-first search when the heuristic is inadmissible. Finally, tabu search algorithms (Glover, 
1989), which maintain a bounded list of states that must not be revisited, have proved effective 
for optimization problems in operations research. 

Simulated annealing was first described by Kirkpatrick, Gelatt, and Vecchi (1983), who 
borrowed the algorithm directly from the Metropolis algorithm used to simulate complex 
systems in statistical physics (Metropolis et al., 1953). Simulated annealing is now a subfield in 
itself, with several hundred papers published every year. 
parallel search The topic of parallel search algorithms was not covered in the chapter, partly because it 

requires a lengthy discussion ofparallel architectures. As parallel computers are becoming widely 
available, parallel search is becoming an important topic in both Al and theoretical computer 
science. A brief introduction to the Al literature can be found in Mahanti and Daniels (1993). 

By far the most comprehensive source on heuristic search algorithms is Pearl's (1984) 
Heuristics text. This book provides especially good coverage of the wide variety of offshoots and 
variations of A*, including rigorous proofs of their formal properties. Kanal and Kumar (1988) 
present an anthology of substantive and important articles on heuristic search. New results on 
search algorithms appear regularly in the journal Artificial Intelligence. 
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EXERCISES 

4.1 Suppose that we run a greedy search algorithm with h{n) = —g(n). What sort of search will 
the greedy search emulate? 

4.2 Come up with heuristics for the following problems. Explain whether they are admissible, 
and whether the state spaces contain local maxima with your heuristic: 

a. The general case of the chain problem (i.e., with an arbitrary goal state) fromExercise 3.15. 

b. Algebraic equation solving (e.g., "solve x 2 y 3 = 3 — xy for v”). 

c. Path planning in the plane with rectangular obstacles (see also Exercise 4.13). 

d. Maze problems, as defined in Chapter 3. 

4.3 Consider the problem of constructing crossword puzzles: fitting words into a grid of 
intersecting horizontal and vertical squares. Assume that a list of words (i.e., a dictionary) is 
provided, and that the task is to fill in the squares using any subset of this list. Go through a 
complete goal and problem formulation for this domain, and choose a search strategy to solve it. 
Specify the heuristic function, if you think one is needed. 

4.4 Sometimes there is no good evaluation function for a problem, but there is a good comparison 
method: a way to tell if one node is better than another, without assigning numerical values to 
either. Show that this is enough to do a best- first search. What properties of best- first search do 
we give up if we only have a comparison method? 

4.5 We saw on page 95 that the straight-line distance heuristic is misleading on the problem of 
going from Iasi to Fagaras. However, the heuristic is perfect on the opposite problem: going from 
Fagaras to Iasi . Are there problems for which the heuristic is misleading in both directions? 

4.6 Invent a heuristic function for the 8-puzzle that sometimes overestimates, and show how it 
can lead to a suboptimal solution on a particular problem. 

4.7 Prove that if the heuristic function h obeys the triangle inequality, then the/-cost along any 
path in the search tree is nondecreasing. (The triangle inequality says that the sum of the costs 
from A to B and B to C must not be less than the cost from A to C directly.) 

4.8 We showed in the chapter that an admissible heuristic heuristic (when combined with 
pathmax) leads to monotonically nondecreasing/ values along any path (i.e., f(successor( n ) J> 
/(«)). Does the implication go the other way? That is, does monotonicity in/imply admissibility 
of the associated hi 

4.9 We gave two simple heuristics for the 8-puzzle: Manhattan distance and misplaced tiles. 
Several heuristics in the literature purport to be better than either of these. (See, for example, Nils- 
son (1971) for additional improvements on Manhattan distance, andMostow and Prieditis (1989) 
for heuristics derived by semimechanical methods.) Test these claims by implementing the 
heuristics and comparing the performance of the resulting algorithms. 

4.10 Would a bidirectional A* search be a good idea? Under what conditions would it be 
applicable? Describe how the algorithm would work. 

L 
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4.11 The travelling salesperson problem (TSP) can be solved using the minimum spanning tree 
(MST) heuristic, which is used to estimate the cost of completing a tour, given that a partial tour 
has already been constructed. The MST cost of a set of cities is the smallest sum of the link costs 
of any tree that connects all the cities. 

a. Show how this heuristic can be derived using a relaxed version of the TSP. 

b. Show that the MST heuristic dominates straight-line distance. 

c. Write a problem generator for instances of the TSP where cities are represented by random 
points in the unit square. 

d. Find an efficient algorithm in the literature for constructing the MST, and use it with an 
admissible search algorithm to solve instances of the TSP. 

4.12 Implement the «-queens problem and solve it using hill-climbing, hill-climbing with 
random restart, and simulated annealing. Measure the search cost and percentage of solved 
problems using randomly generated start states. Graph these against the difficulty of the problems 
(as measured by the optimal solution length). Comment on your results. 

4.13 Consider the problem of finding the shortest path between two points on a plane that has 
convex polygonal obstacles as shown in Figure 4.17. This is an idealization of the problem that 
a robot has to solve to navigate its way around a crowded environment. 

a. Suppose the state space consists of all positions (x,y) in the plane. How many states are 
there? How many paths are there to the goal? 

b. Explain briefly why the shortest path from one polygon vertex to any other in the scene 
must consist of straight-line segments joining some of the vertices of the polygons. Define 
a good state space now. How large is this state space? 

c. Define the necessary functions to implement the search problem, including a successor 
function that takes a vertex as input and returns the set of vertices that can be reached in 
a straight line from the given vertex. (Do not forget the neighbors on the same polygon.) 
Use the straight-line distance for the heuristic function. 

d. Implement any of the admissible algorithms discussed in the chapter. Keep the implemen- 
tation of the algorithm completely independent of the specific domain. Then apply it to 
solve various problem instances in the domain. 



4.14 In this question, we will turn the geometric scene from a simple data structure into a 
complete environment. Then we will put the agent in the environment and have it find its way to 
the goal. 

a. Implement an evaluation environment as described in Chapter 2. The environment should 
behave as follows: 

• The percept at each cycle provides the agent with a list of the places that it can see 
from its current location. Each place is a position vector (with an x and y component) 
giving the coordinates of the place relative to the agent. Thus, if the agent is at ( 1 , 1 ) 
and there is a visible vertex at (7,3), the percept will include the position vector (6,2). 
(It therefore will be up to the agent to find out where it is! It can assume that all 
locations have a different “view.”) 
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• The action returned by the agent will be the vector describing the straight-line path 
it wishes to follow (thus, the relative coordinates of the place it wishes to go). If the 
move does not bump into anything, the environment should "move" the agent and 
give it the percept appropriate to the next place; otherwise it stays put. If the agent 
wants to move (0,0) and is at the goal, then the environment should move the agent 
to a random vertex in the scene. (First pick a random polygon, and then a random 
vertex on that polygon.) 

b. Implement an agent function that operates in this environment as follows: 

• If it does not know where it is, it will need to calculate that from the percept. 

• If it knows where it is and does not have a plan, it must calculate a plan to get home 
to the goal, using a search algorithm. 

• Once it knows where it is and has a plan, it should output the appropriate action from 
the plan. It should say (0,0) once it gets to the goal. 

c. Show the environment and agent operating together. The environment should print out 
some useful messages for the user showing what is going on. 

d. Modify the environment so that 30% of the time the agent ends up at an unintended 
destination (chosen randomly from the other visible vertices if any, otherwise no move at 
all). This is a crude model of the actual motion errors from which both humans and robots 
suffer. Modify the agent so that it always tries to get back on track when this happens. 
What it should do is this: if such an error is detected, first find out where it is and then 
modify its plan to first go back to where it was and resume the old plan. Remember that 
sometimes getting back to where it was may fail also! Show an example of the agent 
successfully overcoming two successive motion errors and still reaching the goal. 

e. Now try two different recovery schemes after an error: head for the closest vertex on 
the original route; and replan a route to the goal from the new location. Compare the 
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performance of the three recovery schemes using a variety of exchange rates between 
search cost and path cost. 

4.15 In this exercise, we will examine hill-climbing in the context of robot navigation, using 
the environment in Figure 4.17 as an example. 

a. Repeat Exercise 4.14 using hill-climbing. Does your agent ever get stuck on a local 
maximum? Is it possible for it to get stuck with convex obstacles? 

b. Construct a nonconvex polygonal environment in which the agent gets stuck. 

c. Modify the hill-climbing algorithm so that instead of doing a depth- 1 search to decide 
where to go next, it does a depth-/: search. It should find the best k-step path and do one 
step along it, and then repeat the process. 

d. Is there some k for which the new algorithm is guaranteed to escape from local maxima? 

4.16 Prove that IDA* returns optimal solutions whenever there is sufficient memory for the 
longest path with cost < /* . Could it be modified along the lines of SMA* to succeed even with 
enough memory for only the shortest solution path? 

4.17 Compare the performance of A*, SMA*, and IDA* on a set of randomly generated 
problems in the 8-puzzle (with Manhattan distance) and TSP (with minimum spanning tree) 
domains. Discuss your results. What happens to the performance of IDA* when a small random 
number is added to the heuristic values in the 8-puzzle domain? 

4.18 Proofs of properties of SMA*: 

a. SMA* abandons paths that fill up memory by themselves but do not contain a solution. 
Show that without this check, SMA" will get stuck in an infinite loop whenever it does not 
have enough memory for the shortest solution path. 

b. Prove that SMA* terminates in a finite space or if there is a finite path to a goal. The proof 
will work by showing that it can never generate the same tree twice. This follows from the 
fact that between any two expansions of the same node, the node's parent must increase its 
/-cost. We will prove this fact by a downward induction on the depth of the node. 

(i) Show that the property holds for any node at depth d = MAX. 

(ii) Show that if it holds for all nodes at depth d + 1 or more, it must also hold for all 
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GAME PLAYING 



In which we examine the problems that arise when we try to plan ahead in a world 
that includes a hostile agent. 



5.1 INTRODUCTION: GAMES AS SEARCH PROBLEMS 



Games have engaged the intellectual faculties of humans — sometimes to an alarming degree — for 
as long as civilization has existed. Board games such as chess and Go are interesting in part 
because they offer pure, abstract competition, without the fuss and bother of mustering up two 
armies and going to war. It is this abstraction that makes game playing an appealing target of AI 
research. The state of a game is easy to represent, and agents are usually restricted to a fairly 
. -V . small number of well-defined actions. That makes game playing an idealization of worlds in 

dP which hostile agents act so as to diminish one 's well-being. Less abstract games, such as croquet 

or football, have not attracted much interest in the AI community. 

Game playing is also one of the oldest areas of endeavor in artificial intelligence. In 1950, 
almost as soon as computers became programmable, the first chess programs were written by 
Claude Shannon (the inventor of information theory) and by Alan Turing. Since then, there has 
been steady progress in the standard of play, to the point where current systems can challenge 
the human world champion without fear of gross embarrassment. 

Early researchers chose chess for several reasons. A chess-playing computer would be an 
existence proof of a machine doing something thought to require intelligence. Furthermore, the 
simplicity of the rules, and the fact that the world state is fully accessible to the program 1 means 
that it is easy to represent the game as a search through a space of possible game positions. The 
computer representation of the game actually can be correct in every relevant detail — unlike the 
representation of the problem of fighting a war, for example. 

' Recall from Chapter 2 that accessible means that the agent can perceive everything there is to know about the 
environment. In game theory, chess is called a game of perfect information. 
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The presence of an opponent makes the decision problem somewhat more complicated 
than the search problems discussed in Chapter 3. The opponent introduces uncertainty, because 
one never knows what he or she is going to do. In essence, all game-playing programs must deal 
with the contingency problem defined in Chapter 3. The uncertainty is not like that introduced, 
say, by throwing dice or by the weather. The opponent will try as far as possible to make the least 
benign move, whereas the dice arid the weather are assumed (perhaps wrongly ! ) to be indifferent 
to the goals of the agent. This complication is discussed in Section 5.2. 

But what makes games really different is that they are usually much too hard to solve. 
Chess, for example, has an average branching factor of about 35, and games often go to 50 
moves by each player, so the search tree has about 35 100 nodes (although there are "only" about 
10 40 different legal positions). Tic-Tac-Toe (noughts and crosses) is boring for adults precisely 
because it is easy to determine the right move. The complexity of games introduces a completely 
new kind of uncertainty that we have not seen so far; the uncertainty arises not because there is 
missing information, but because one does not have time to calculate the exact consequences of 
any move. Instead, one has to make one's best guess based on past experience, and act before 
one is sure of what action to take. In this respect, games are much more like the real world than 
the standard search problems we have looked at so far. 

Because they usually have time limits, games also penalize inefficiency very severely. 
Whereas an implementation of A* search that is 10% less efficient will simply cost a little bit 
extra to run to completion, a chess program that is 1 0% less effective in using its available time 
probably will be beaten into the ground, other things being equal. Game-playing research has 
therefore spawned a number of interesting ideas on how to make the best use of time to reach 
good decisions, when reaching optimal decisions is impossible. These ideas should be kept in 
mind throughout the rest of the book, because the problems of complexity arise in every area of 
AI. We will return to them in more detail in Chapter 16. 

We begin our discussion by analyzing how to find the theoretically best move. We then 
look at techniques for choosing a good move when time is limited. Pruning allows us to ignore 
portions of the search tree that make no difference to the final choice, and heuristic evaluation 
functions allow us to approximate the true utility of a state without doing a complete search. 
Section 5.5 discusses games such as backgammon that include an element of chance. Finally, we 
look at how state-of-the-art game-playing programs succeed against strong human opposition. 



5.2 Perfect DECISIONS IN TWO-PERSON GAMES 



We will consider the general case of a game with two players, whom we will call MAX and MIN, 
for reasons that will soon become obvious. MAX moves first, and then they take turns moving 
until the game is over. At the end of the game, points are awarded to the winning player (or 
sometimes penalties are given to the loser). A game can be formally defined as a kind of search 
problem with the following components: 

• The initial state, which includes the board position and an indication of whose move it is. 

• A set of operators, which define the legal moves that a player can make. 
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TERMINAL TEST • A terminal test, which determines when the game is over. States where the game has 

ended are called terminal states. 

PAYOFF FUNCTION • A utility function (also called a payoff function), which gives a numeric value for the 

outcome of a game. In chess, the outcome is a win, loss, or draw, which we can represent 
by the values +1, —1. or 0. Some games have a wider variety of possible outcomes; for 
example, the payoffs in backgammon range from +192 to —192. 

If this were a normal search problem, then all MAX would have to do is search for a sequence 
of moves that leads to a terminal state that is a winner (according to the utility function), and 
then go ahead and make the first move in the sequence. Unfortunately, MIN has something to say 
STRATEGY about it. MAX therefore must find a strategy that will lead to a winning terminal state regardless 

of what MIN does, where the strategy includes the correct move for MAX for each possible move 
by MIN. We will begin by showing how to find the optimal (or rational) strategy, even though 
normally we will not have enough time to compute it. 

Figure 5.1 shows part of the search tree forthe game ofTic-Tac-Toe. From the initial state, 
MAX has a choice of nine possible moves. Play alternates between MAX placing x's and MIN 
placing o's until we reach leaf nodes corresponding to terminal states: states where one player 
has three in a row or all the squares are filled. The number on each leaf node indicates the utility 
value of the terminal state from the point of view of MAX; high values are assumed to be good 
for MAX and bad for MIN (which is how the players get their names). It is MAX'S job to use the 
search tree (particularly the utility of terminal states) to determine the best move. 

Even a simple game like Tic-Tac-Toe is too complex to show the whole search tree, so we 
will switch to the absolutely trivial game in Figure 5.2. The possible moves for MAX are labelled 
A\, A2, and A3. The possible replies to A\ for MIN are An, A\2, A 13 , and so on. This particular 
game ends after one move each by MAX and MIN. (In game parlance, we say this tree is one move 
ply deep, consisting of two half-moves or two ply.) The utilities of the terminal states in this game 

range from 2 to 14. 

m nimax The minimax algorithm is designed to determine the optimal strategy for MAX, and thus 

to decide what the best first move is. The algorithm consists of five steps: 

• Generate the whole game tree, all the way down to the terminal states. 

• Apply the utility function to each terminal state to get its value. 

• Use the utility of the terminal states to determine the utility of the nodes one level higher 
up in the search tree. Consider the leftmost three leaf nodes in Figure 5.2. In the V node 
above it, MIN has the option to move, and the best MIN can do is choose A\ \ , which leads 
to the minimal outcome, 3. Thus, even though the utility function is not immediately 
applicable to this V node, we can assign it the utility value 3, under the assumption that 
MIN will do the right thing. By similar reasoning, the other two V nodes are assigned the 
utility value 2 . 

• Continue backing up the values from the leaf nodes toward the root, one layer at a time. 

• Eventually, the backed-up values reach the top of the tree; at that point, MAX chooses the 
move that leads to the highest value. In the topmost A node of Figure 5.2, MAX has a choice 
of three moves that will lead to states with utility 3, 2, and 2, respectively. Thus, MAX's 

minimax decision best opening move is A] 1 . This is called the mininiax decision, because it maximizes the 

utility under the assumption that the opponent will play perfectly to minimize it. 
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Figure 5.2 A two-ply game tree as generated by the minimax algorithm. The A nodes ars 
moves by MAX and the V nodes aremoves by MIN. The terminal nodes show the utility valuefor 
MAX computed by the utility function (i.e., by the rules of the game), whereas the utilities of the 
other nodes are computed by the minimax algorithm from the utilities of their successors. MAX'S 
best move is A\ and min’s best reply is AW. 
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Figure 5.3 shows a more formal description of the minimax algorithm. The top level func- 
tion, MINIMAX-DECISION, selects from the available moves, which are evaluated in turn by the 
Minimax- Value function. 

If the maximum depth of the tree is m, and there are b legal moves at each point, then 
the time complexity of the minimax algorithm is 0{b m ). The algorithm is a depth-first search 
(although here the implementation is through recursion rather than using a queue of nodes), so 
its space requirements are only linear in m and b. For real games, of course, the time cost is 
totally impractical, but this algorithm serves as the basis for more realistic methods and for the 
mathematical analysis of games. 



function MlNJMAX-DEClsiON(gamc) returns an operator 

for each op in OPERATORS[game] do 

Value[o/?1 — Mi\lmax-Value(Apply(o/?. game), game) 

end 

return the op with the highest VALUEfop] 



function VI in [MAX- Value) .state, game) returns a utility value 

if T ERMlNAL-TEST[game |(.state ) then 
return \} r X\UTi\game\(state) 
else if MAX is to move in state then 

return the highest Mini max- Value of SuccESSORS(.state ) 

else 

return the lowest Minimax- Value of SucCESSORS(ifafe) 



Figure 5.3 An algorithm for calculating minimax decisions. It returns the operator that 
corresponding to the best possible move, that is, the move that leads to the outcome with the 
best utility, under the assumption that the opponent plays to minimize utility. The function 
MINIMAX-VALUE goes through the whole game tree, all the way to the leaves, to determine the 
backed-up value of a state. 



5.3 IMPERFECT DECISIONS 



The minimax algorithm assumes that the program has time to search all the way to terminal 
states, which is usually not practical. Shannon's original paper on chess proposed that instead of 
going all the way to terminal states and using the utility function, the program should cut off the 
search earlier and apply a heuristic evaluation function to the leaves of the tree. In other words, 
the suggestion is to alter minimax in two ways: the utility function is replaced by an evaluation 
function EVAL, and the terminal test is replaced by a cutoff test CUTOFF-TEST. 
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Evaluation functions 

An evaluation function returns an estimate of the expected utility of the game from a given 
position. The idea was not new when Shannon proposed it. For centuries, chess players (and, of 
course, aficionados of other games) have developed ways ofjudging the winning chances of each 
side based on easily calculated features of a position. For example, introductory chess books 
material VALUE give an approximate material value for each piece: each pawn is worth 1, a knight or bishop 

is worth 3, a rook 5, and the queen 9. Other features such as "good pawn structure" and "king 
safety" might be worth half a pawn, say. All other things being equal, a side that has a secure 
material advantage of a pawn or more will probably win the game, and a 3-point advantage is 
sufficient for near-certain victory. Figure 5.4 shows four positions with their evaluations. 

It should be clear that the performance of a game-playing program is extremely dependent 
on the quality of its evaluation function. If it is inaccurate, then it will guide the program toward 
positions that are apparently "good," but in fact disastrous. How exactly do we measure quality? 

First, the evaluation function must agree with the utility function on terminal states. Second, 
it must not take too long! (As mentioned in Chapter 4, if we did not limit its complexity, then it 
could call minimax as a subroutine and calculate the exact value of the position.) Hence, there is 
a trade-off between the accuracy of the evaluation function and its time cost. Third, an evaluation 
function should accurately reflect the actual chances of winning. 

One might well wonder about the phrase "chances of winning." After all, chess is not a 
game of chance. But if we have cut off the search at a particular nonterminal state, then we do not 
know what will happen in subsequent moves. For concreteness, assume the evaluation function 
counts only material value. Then, in the opening position, the evaluation is 0, because both sides 
have the same material. All the positions up to the first capture will also have an evaluation of 
0. If MAX manages to capture a bishop without losing a piece, then the resulting position will 
have an evaluation value of 3. The important point is that a given evaluation value covers many 
different positions — all the positions where MAX is up by a bishop are grouped together into 
a category that is given the label "3." Now we can see how the word "chance" makes sense: 
the evaluation function should reflect the chance that a position chosen at random from such a 
category leads to a win (or draw or loss), based on previous experience. 2 

This suggests that the evaluation function should be specified by the rules of probability: 
if position A has a 100% chance of winning, it should have the evaluation 1 .00, and if position B 
has a 50% chance of winning, 25% of losing, and 25% of being a draw, its evaluation should be 
+1 x .50 + -1 x .25 + 0 x .25 = .25. But in fact, we need not be this precise; the actual numeric 
values of the evaluation function are not important, as long as A is rated higher than B. 

The material advantage evaluation function assumes that the value of a piece can be judged 
independently of the other pieces present on the board. This kind of evaluation function is called 
a weighted linear function, because it can be expressed as 

Wl/l + wrf 2 + • • • + Wjn 

where the w’s are the weights, and the/’s are the features of the particular position. The w's 
would be the values ofthe pieces (1 forpawn, 3 for bishop, etc.), and the/’s wouldbe the numbers 

2 Techniques for automatically constructing evaluation functions with this property are discussed in Chapter 18. In 
assessing the value of a category, more normally occurring positions should be given more weight. 
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Cutting off search 

The most straightforward approach to controlling the amount of search is to set a fixed depth 
limit, so that the cutoff test succeeds for all nodes at or below depth d. The depth is chosen so 
that the amount of time used will not exceed what the rules of the game allow. A slightly more 
robust approach is to apply iterative deepening, as defined in Chapter 3. When time runs out, the 
program returns the move selected by the deepest completed search. 

These approaches can have some disastrous consequences because of the approximate 
nature of the evaluation function. Consider again the simple evaluation function for chess based 
on material advantage. Suppose the program searches to the depth limit, reaching the position 
shown in Figure 5.4(d). According to the material function, white is ahead by a knight and 
therefore almost certain to win. However, because it is black's move, white's queen is lost 
because the black knight can capture it without any recompense for white. Thus, in reality the 
position is won for black, but this can only be seen by looking ahead one more ply. 

Obviously, a more sophisticated cutoff test is needed. The evaluation function should only 
quiescent be applied to positions that are quiescent, that is, unlikely to exhibit wild swings in value in the 

near future. In chess, for example, positions in which favorable captures can be made are not 
quiescent for an evaluation function that just counts material. Nonquiescent positions can be 
expanded further until quiescent positions are reached. This extra search is called a quiescence 
search ENCE search; sometimes it is restricted to consider only certain types of moves, such as capture moves, 

that will quickly resolve the uncertainties in the position. 
horizon problem The horizon problem is more difficult to eliminate. It arises when the program is facing 

a move by the opponent that causes serious damage and is ultimately unavoidable. Consider the 
chess game in Figure 5.5. Black is slightly ahead in material, but if white can advance its pawn 
from the seventh row to the eighth, it will become a queen and be an easy win for white. Black can 
forestall this for a dozen or so ply by checking white with the rook, but inevitably the pawn will 
become a queen. The problem with fixed-depth search is that it believes that these stalling moves 
have avoided the queening move — we say that the stalling moves push the inevitable queening 
move "over the horizon" to a place where it cannot be detected. At present, no general solution 
has been found for the horizon problem. 



5.4 Alpha -RetaPRUNING 



Let us assume we have implemented a minimax search with a reasonable evaluation function for 
chess, and a reasonable cutoff test with a quiescence search. With a well-written program on an 
ordinary computer, one can probably search about 1000 positions a second. How well will our 
program play? In tournament chess, one gets about 1 50 seconds per move, so we can look at 
150,000 positions. In chess, the branching factor is about 35, so our program will be able to look 
ahead only three or four ply, and will play at the level of a complete novice! Even average human 
players can make plans six or eight ply ahead, so our program will be easily fooled. 

Fortunately, it is possible to compute the correct minimax decision without looking at every 
node in the search tree. The process of eliminating a branch of the search tree from consideration 
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Black to move 



Figure 5.5 The horizon problem. A series of checks by the black rook forces the inevitable 
queening move by white "over the horizon" and makes this position look like a slight advantage 
for black, when it is really a sure win for white. 



pruning without examining it is called pruning the search tree. The particular technique we will examine 

pruning™ is called alpha-beta pruning. When applied to a standard minimax tree, it returns the same move 

as minimax would, but prunes away branches that cannot possibly influence the final decision. 

Consider the two-ply game tree from Figure 5.2, shown again in Figure 5.6. The search 
proceeds as before: A\ then A\\ A\2, A13, and the node under Aj gets minimax value 3. Now 
we followA2, andA2i . which has value 2. At this point, we realize that if MAX plays A2, MIN has 
the option of reaching a position worth 2, and some other options besides. Therefore, we can say 
already that move A2 is worth at most 2 to MAX. Because we already know that moveA\ is worth 
3, there is no point at looking further under A2. In other words, we can prune the search tree at 
this point and be confident that the pruning will have no effect on the outcome. 

The general principle is this. Consider a node n somewhere in the tree (see Figure 5.7), 
such that Player has a choice of moving to that node. If Player has a better choice m either at the 
. ' . parent node of n, or at any choice point further up, then n will never be reached in actual play. 

I AS So once we have found out enough about n (by examining some of its descendants) to reach this 

conclusion, we can prune it. 

Remember that minimax search is depth-first, so at any one time we just have to consider 
the nodes along a single path in the tree. Let a be the value of the best choice we have found so 
far at any choice point along the path for MAX, and 9 be the value of the best (i.e., lowest- value) 
choice we have found so far at any choice point along the path for MIN. Alpha-beta search updates 
the value of a and 8 as it goes along, and prunes a subtree (i.e., terminates the recursive call) as 
soon as it is known to be worse than the current a or 13 value. 

The algorithm description in Figure 5.8 is divided into a MAX- VALUE function and a 
Min-Value function. These apply to MAX nodes and min nodes, respectively, but each does the 
same thing: return the minimax value of the node, except for nodes that are to be pruned (in 
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which case the returned value is ignored anyway). The alpha-beta search function itself is just a 
copy of the Max- Value function with extra code to remember and return the best move found. 



Effectiveness of alpha-beta pruning 

The effectiveness of alpha-beta depends on the ordering in which the successors are examined. 
This is clear from Figure 5.6, where we could not prune A3 at all because A31 and A32 (the worst 
moves from the point of view of MIN) were generated first. This suggests it might be worthwhile 
to try to examine first the successors that are likely to be best. 

If we assume that this can be done, 3 then it turns out that alpha-beta only needs to examine 
Offc^fnodes to pick the best move, instead of 0(b d ) with minimax. This means that the effective 
branching factor is \fb instead of b — for chess, 6 instead of 35. Put another way, this means 



3 Obviously, it 



: be done perfectly, otherwise the ordering function could be used to play a perfect game! 
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function Max- Value {state game, a, /l) returns the minimax value of state 
inputs: state, current state in game 
game, game description 

a, the best score for MAX along the path to state 
/?, the best score for MIN along the path to state 

if CuTOFF-TEST(s?a?e) then return EVAL(state) 
for each s in SucCESSORS(s?a?e) do 

a <- Max(q , Min- VALUE! . s, game, a, /?)) 

if a > ft then return/? 
end 

return a 



function MiN-VALUE(.v?a?e, game, a, ft) returns the minimax value of state 

if CuTOFF-T EST(state ) then return EVAL (state) 
for each s in SucCESSORS(itote) do 

ft — Min(/?, Max -Valuer, game, a, /?)) 

it ft < a then return a 

end 

return/? 



Figure 5.8 The alpha-beta search algorithm. It does the same computation as a normal 
minimax, but prunes the search tree. 



that alpha-beta can look ahead twice as far as minimax for the same cost. Thus, by generating 
150,000 nodes in the time allotment, a program can look ahead eight ply instead of four. By 
thinking carefully about which computations actually affect the decision, we are able to transform 
a program from a novice into an expert. 

The effectiveness of alpha-beta pruning was first analyzed in depth by Knuth and Moore 
(1975). As well as the best case described in the previous paragraph, they analyzed the 
case in which successors are ordered randomly. It turns out that the asymptotic complexity 
is 0((b/logb) d ), which seems rather dismal because the effective branching factor b/logb is not 
much less than b itself. On the other hand, the asymptotic formula is only accurate for b > 1000 
or so — in other words, not for any games we can reasonably play using these techniques. For 
reasonable b, the total number of nodes examined will be roughly 0(b 3d/ 4 ). In practice, a fairly 
simple ordering function (such as trying captures first, then threats, then forward moves, then 
backward moves) gets you fairly close to the best-case result rather than the random result. 
Another popular approach is to do an iterative deepening search, and use the backed-up values 
from one iteration to determine the ordering of successors in the next iteration. 

It is also worth noting that all complexity results on games (and, in fact, on search problems 
in general) have to assume an idealized tree model in order to obtain their results. For example, 
the model used for the alpha-beta result in the previous paragraph assumes that all nodes have the 
same branching factor b; that all paths reach the fixed depth limit d\ and that the leaf evaluations 
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are randomly distributed across the last layer of the tree. This last assumption is seriously flawed: 
for example, if a move higher up the tree is a disastrous blunder, then most of its descendants 
will look bad for the player who made the blunder. The value of a node is therefore likely to 
be highly correlated with the values of its siblings. The amount of correlation depends very 
much on the particular game and indeed the particular position at the root. Hence, there is 
an unavoidable component of empirical science involved in game-playing research, eluding the 
power of mathematical analysis. 



5.5 GAMES THAT INCL UDE AN ELEMENT OF C HANCE 



In real life, unlike chess, there are many unpredictable external events that put us into unforeseen 
situations. Many games mirror this unpredictability by including a random element such as 
throwing dice. In this way, they take us a step nearer reality, and it is worthwhile to see how this 
affects the decision-making process. 

Backgammon is a typical game that combines luck and skill. Dice are rolled at the 
beginning of a player's turn to determine the set of legal moves that is available to the player. In 
the backgammon position of Figure 5.9, white has rolled a 6-5, and has four possible moves. 

Although white knows what his or her own legal moves are, white does not know what 
black is going to roll, and thus does not know what black's legal moves will be. That means white 
cannot construct a complete game tree of the sort we saw in chess and Tic-Tac-Toe. A game tree 
CHANCE NODES in backgammon must include chance nodes in addition to MAX and min nodes. Chance nodes 
are shown as circles in Figure 5.10. The branches leading from each chance node denote the 
possible dice rolls, and each is labelled with the roll and the chance that it will occur. There are 
36 ways to roll two dice, each equally likely; but because a 6-5 is the same as a 5-6, there are 
only 21 distinct rolls. The six doubles (1-1 through 6-6) have a 1/36 chance of coming up, the 
other fifteen distinct rolls a 1/18 chance. 

The next step is to understand how to make correct decisions. Obviously, we still want 
to pick the move from A\ , ... ,A„ that leads to the best position. However, each of the possible 
positions no longer has a definite minimax value (which in deterministic games was the utility 
EXPECTED VALUE of the leaf reached by best play). Instead, we can only calculate an average or expected value, 
where the average is taken over all the possible dice rolls that could occur. 

It is straightforward to calculate expected values of nodes. For terminal nodes, we use the 
utility function, just like in deterministic games. Going one step up in the search tree, we hit a 
chance node. In Figure 5.10, the chance nodes are circles; we will consider the one labelled C. 
Let di be a possible dice roll, and P(dj) be the chance or probability of obtaining that roll. For 
each dice roll, we calculate the utility of the best move for MIN, and then add up the utilities, 
weighted by the chance that the particular dice roll is obtained. If we let S(C, di) denote the set 
of positions generated by applying the legal moves for dice roll P(dj) to the position at C, then 
expectimax VALUE we can calculate the so-called expectimax value of C using the formula 

expectimax(C) = P(d,) m<vt. s< z S (c,d,)(utility(s)) 



Figure 5.9 A typical backgammon position. The aim of the game is to move all one's pieces 
off the board. White moves clockwise toward 25, and black moves counterclockwise toward 0. A 
piece can move to any position except one where there are two or more of the opponent's pieces. 
If it moves to a position with one opponent piece, that piece is captured and has to start its journey 
again from the beginning. In this position, white has just rolled 6-5 and has four legal moves: 
(5- 10,5-1 1), (5-1 1 , 1 9-24), (5-10,10- 1 6), and (5-1 1 , 1 1 - 1 6). 

MAX 



DICE 



MIN 



DICE 



MAX 



TERMINAL 

Figure 5.10 Schematic game tree for a backgammon position. 
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This gives us the expected utility of the position at C assuming best play. Going up one more ;n 

level to the MIN nodes (v in Figure 5.10), we can now apply the normal minimax-value formula. J 

because we have assigned utility values to all the chance nodes. We then move up to chance node || 

expectimin VALUE B. where we can compute the expectimin value using a formula that is analogous to expectimax. j| 

This process can be applied recursively all the way up the tree, except at the top level j| 

where the dice roll is already known. To calculate the best move, then, we simply replace H 

MINIMAX- VALUE in Figure 5.3 by EXPECTIMINIM AX- VALUE, the implementation of which we jl 

leave as an exercise. H 



Position evaluation in games with chance nodes 

As with minimax, the obvious approximation to make with expectiminimax is to cut off search 
at some point and apply an evaluation function to the leaves. One might think that evaluation 
functions for games such as backgammon are no different, in principle, from evaluation functions 
for chess — they shouldjust give higher scores to better positions. 

In fact, the presence of chance nodes means one has to be more careful about what the 
evaluation values mean. Remember that for minimax, any order-preserving transformation of 
the leaf values does not affect the choice of move. Thus, we can use either the values 1 , 2, 3, 4 
or the values 1, 20, 30, 400, and get the same decision. This gives us a good deal of freedom in 
designing the evaluation function: it will work fine as long as positions with higher evaluations 
lead to wins more often, on average. 

With chance nodes, we lose this freedom. Figure 5.11 shows what happens: with leaf 
values 1, 2, 3, 4, move A\ is best; with leaf values 1, 20, 30, 400, move Ai is best. Hence, 
the program behaves totally differently if we make a change in the scale of evaluation values ! 
It turns out that to avoid this sensitivity, the evaluation function can be only a positive linear 
transformation of the likelihood of winning from a position (or, more generally, of the expected 
utility of the position). This is an important and general property of situations in which uncertainty 
is involved, and we discuss it further in Chapter 16. 



Complexity of expectiminimax 

If the program knew in advance all the dice rolls that would occur for the rest ofthe game, solving 
a game with dice would be just like solving a game without dice, which minimax does in 0(b m ) 
time. Because expectiminimax is also considering all the possible dice-roll sequences, it will 
take 0{b m n m ), where n is the number of distinct rolls. 

Even if the depth of the tree is limited to some small depth d , the extra cost compared to 
minimax makes it unrealistic to consider looking ahead very far in games such as backgammon, 
where n is 21 and b is usually around 20, but in some situations can be as high as 4000. Two ply 
is probably all we could manage. 

Another way to think about the problem is this: the advantage of alpha-beta is that it ignores 
future developments that just are not going to happen, given best play. Thus, it concentrates on 
likely occurrences. In games with dice, there are no likely sequences of moves, because for those 
moves to take place, the dice would first have to come out the right way to make them legal. 

L 
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This is a general problem whenever uncertainty enters the picture: the possibilities are multiplied 
enormously, and forming detailed plans of action becomes pointless because the world probably 
will not play along. 

No doubt it will have occurred to the reader that perhaps something like alpha-beta pruning 
could be applied to game trees with chance nodes. It turns out that it can, with a bit of ingenuity. 
Consider the chance node C in Figure 5.10, and what happens to its value as we examine and 
evaluate its children; the question is, is it possible to find an upper bound on the value of C 
before we have looked at all its children? (Recall that this is what alpha-beta needs in order to 
prune a node and its subtree.) At first sight, it might seem impossible, because the value of C is 
the average of its children's values, and until we have looked at all the dice rolls, this average 
could be anything, because the unexamined children might have any value at all. But if we put 
boundaries on the possible values of the utility function, then we can arrive at boundaries for the 
average. For example, if we say that all utility values are between +1 and —1, then the value of 
leaf nodes is bounded, and in turn we can place an upper bound on the value of a chance node 
without looking at all its children. Designing the pruning process is a little bit more complicated 
than for alpha-beta, and we leave it as an exercise. 



5.6 STATE-OF-THE-ART GAME PROGRAMS 



Designing game-playing programs has a dual purpose: both to better understand how to choose 
actions in complex domains with uncertain outcomes and to develop high-performance systems 
for the particular game studied. In this section, we examine progress toward the latter goal. 
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Chess 

Chess has received by far the largest share of attention in game playing. Although not meeting 
the promise made by Simon in 1957 that within 10 years, computers would beat the human world 
champion, they are now within reach of that goal. In speed chess, computers have defeated the 
world champion, Gary Kasparov, In both 5-minute and 25-minute games, but in full tournament 
games are only ranked among the top lOOplayers worldwide at the time of writing. Figure 5.12 
shows the ratings of human and computer champions over the years. It is tempting to try to 
extrapolate and see where the lines will cross. 

Progress beyond a mediocre level was initially very slow: some programs in the early 
1 970s became extremely complicated, with various kinds of tricks for eliminating some branches 
of search, generating plausible moves, and so on, but the programs that won the ACM North 
American Computer Chess Championships (initiated in 1970)tendedtouse straightforwardalpha- 
beta search, augmented with book openings and infallible endgame algorithms. (This offers an 
interesting example of how high performance requires a hybrid decision-making architecture to 
implement the agent function.) 

The first realjump in performance came not from better algorithms or evaluation functions, 
but from hardware. Belle, the first special-purpose chess computer (Condon and Thompson, 
1982), used custom integrated circuits to implement move generation and position evaluation, 
enabling it to search several million positions to make a single move. Belle's rating was around 
2250, on a scale where beginning humans are 1000 and the world champion around 2750; it 
became the first master-level program. 

The HITECH system, also a special-purpose computer, was designed by former world 
correspondence champion Hans Berliner and his student Carl Ebeling to allow rapid calculation 
of very sophisticated evaluation functions. Generating about 10 million positions per move 
and using probably the most accurate evaluation of positions yet developed, HITECH became 




Figure 5.12 Ratings of human and machine chess champions. 
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computer world champion in 1985, and was the first program to defeat a human grandmaster, 
Arnold Denker, in 1987. At the time it ranked among the top 800 human players in the world. 

The best current system is Deep Thought 2. It is sponsored by IBM, which hired part of the 
team that built the Deep Thought system at Carnegie Mellon University. Although Deep Thought 
2 uses a simple evaluation function, it examines about half a billion positions per move, allowing 
it to reach depth 1 0 or 11, with a special provision to follow lines of forced moves still further 
(it once found a 37-move checkmate). In February 1993, Deep Thought 2 competed against the 
Danish Olympic team and won, 3-1, beating one grandmaster and drawing against another. Its 
FIDE rating is around 2600, placing it among the top 100 human players. 

The next version of the system, Deep Blue, will use a parallel array of 1024 custom VLSI 
chips. This will enable it to search the equivalent of one billion positions per second (100-200 
billion per move) and to reach depth 14. A 1 0-processorversion is due to play the Israeli national 
team (one of the strongest in the world) in May 1995, and the full-scale system will challenge 
the world champion shortly thereafter. 



Checkers or Draughts 

Beginning in 1952, Arthur Samuel of IBM, working in his spare time, developed a checkers 
program that learned its own evaluation function by playing itself thousands of times. We 
describe this idea in more detail in Chapter 20. Samuel's program began as a novice, but after 
only a few days' self-play was able to compete on equal terms in some very strong human 
tournaments. When one considers that Samuel's computing equipment (an IBM 704) had 10,000 
words of main memory, magnetic tape for long-term storage, and a cycle time of almost a 
millisecond, this remains one of the great feats of AI. 

Few other people attempted to do better until Jonathan Schaefferand colleagues developed 
Chinook, which runs on ordinary computers using alpha-beta search, but uses several techniques, 
including perfect solution databases for all six-piece positions, that make its endgame play 
devastating. Chinook won the 1992 U.S. Open, and became the first program to officially 
challenge for a real world championship. It then ran up against a problem, in the form of Marion 
Tinsley. Dr. Tinsley had been world champion for over 40 years, losing only three games in all that 
time. In the first match against Chinook, Tinsley suffered his fourth and fifth losses, but won the 
match21.5-18.5. More recently, the world championship match in August 1994between Tinsley 
and Chinook ended prematurely when Tinsley had to withdraw for health reasons. Chinook 
became the official world champion. 



Othello 

Othello, also called Reversi, is probably more popular as a computer game than as a board game. 
It has a smaller search space than chess, usually 5 to 15 legal moves, but evaluation expertise 
had to be developed from scratch. Even so, Othello programs on normal computers are far better 
than humans, who generally refuse direct challenges in tournaments. 
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Backgammon 

As mentioned before, the inclusion of uncertainty from dice rolls makes search an expensive 
luxury in backgammon. The first program to make a serious impact, BKG, used only a one-ply 
search but a very complicated evaluation function. In an informal match in 1980, it defeated the 
human world champion 5-1, but was quite lucky with the dice. Generally, it plays at a strong 
amateur level. 

More recently, Gerry Tesauro (1992) combined Samuel's learning method with neural 
network techniques (Chapter 19) to develop a new evaluation function. His program is reliably 
ranked among the top three players in the world. 

Go 

Go is the most popular board game in Japan, requiring at least as much discipline from its 
professionals as chess. The branching factor approaches 360, so that regular search methods are 
totally lost. Systems based on large knowledge bases of rules for suggesting plausible moves 
seem to have some hope, but still play very poorly. Particularly given the $2,000,000 prize for the 
first program to defeat a top-level player, Go seems like an area likely to benefit from intensive 
investigation using more sophisticated reasoning methods. 



5.7 DISCUSSION 



Because calculating optimal decisions in games is intractable in most cases, all algorithms 
must make some assumptions and approximations. The standard approach, based on minimax, 
evaluation functions, and alpha-beta, isjust one way to do this. Probably because it was proposed 
so early on, it has been developed intensively and dominates other methods in tournament 
play. Some in the field believe that this has caused game playing to become divorced from the 
mainstream of AI research, because the standard approach no longer provides much room for 
new insight into general questions of decision making. In this section, we look at the alternatives, 
considering how to relax the assumptions and perhaps derive new insights. 

First, let us consider minimax. Minimax is an optimal method for selecting a move from 
a given search tree provided the leaf node evaluations are exactly correct. In reality, evaluations 
are usually crude estimates of the value of a position, and can be considered to have large 
errors associated with them. Figure 5.13 shows a two-ply game tree for which minimax seems 
inappropriate. Minimax suggests taking the right-hand branch, whereas it is quite likely that 
true value of the left-hand branch is higher. The minimax choice relies on the assumption that 
all of the nodes labelled with values 100, 101, 102, and 100 are actually better than the node 
labelled with value 99. One way to deal with this problem is to have an evaluation that returns a 
probability distribution over possible values. Then one can calculate the probability distribution 
for the parent's value using standard statistical techniques. Unfortunately, the values of sibling 
nodes are usually highly correlated, so this can be an expensive calculation and may require 
detailed correlation information that is hard to obtain. 
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Next, we consider the search algorithm that generates the tree. The aim of an algorithm 
designer is to specify a computation that completes in a timely manner and results in a good move 
choice. The most obvious problem with the alpha-beta algorithm is that it is designed not just to 
select a good move, but also to calculate the values of all the legal moves. To see why this extra 
information is unnecessary, consider a position in which there is only one legal move. Alpha-beta 
search still will generate and evaluate a large, and totally useless, search tree. Of course, we 
can insert a test into the algorithm, but this merely hides the underlying problem — many of the 
calculations done by alpha-beta are largely irrelevant. Having only one legal move is not much 
different from having several legal moves, one of which is fine and the rest of which are obviously 
disastrous. In a "clear-favorite" situation like this, it would be better to reach a quick decision 
after a small amount of search than to waste time that could be better used later for a more 
problematic position. This leads to the idea of the utility of a node expansion. A good search 
algorithm should select node expansions of high utility — that is, ones that are likely to lead to the 
discovery of a significantly better move. If there are no node expansions whose utility is higher 
than their cost (in terms of time), then the algorithm should stop searching and make a move. 
Notice that this works not only for clear-favorite situations, but also for the case of symmetrical 
moves, where no amount of search will show that one move is better than another. 

This kind of reasoning about what computations to do is called metareasoning (reasoning 
about reasoning). It applies not just to game playing, but to any kind of reasoning at all. All 
computations are done in the service of trying to reach better decisions, all have costs, and 
all have some likelihood of resulting in a certain improvement in decision quality. Alpha-beta 
incorporates the simplest kind of metareasoning, namely, a theorem to the effect that certain 
branches of the tree can be ignored without loss. It is possible to do much better. In Chapter 16, 
we will see how these ideas can be made precise and implementable. 

Finally, let us reexamine the nature of search itself. Algorithms for heuristic search and 
for game playing work by generating sequences of concrete states starting from the initial state 
and then applying an evaluation function. Clearly, this is not how humans play games. In 
chess, one often has a particular goal in mind — forexample, trapping the opponent's queen — and 
can use this to selectively generate plausible plans for achieving it. This kind of goal-directed 
reasoning or planning sometimes eliminates combinatorial search altogether (see Part IV). David 
Wilkins' (1980) PARADISE is the only program to have used goal-directed reasoning successfully 
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in chess: it was capable of solving some chess problems requiring an 18-move combination. As 
yet, however, there is no good understanding of how to combine the two kinds of algorithm into 
a robust and efficient system. Such a system would be a significant achievement not just for 
game-playing research, but also for AI research in general, because it would be much more likely 
to apply to the problem faced by a general intelligent agent. 

5.8 Summary 



Games are fascinating, and writing game-playing programs perhaps even more so. We might 
say that game playing is to AI as Grand Prix motor racing is to the car industry: although the 
specialized task and extreme competitive pressure lead one to design systems that do not look 
much like your garden-variety, general-purpose intelligent system, a lot of leading-edge concepts 
and engineering ideas come out of it. On the other hand, just as you would not expect a Grand 
Prix racing car to perform well on a bumpy dirt road, you should not expect advances in game 
playing to translate immediately into advances in less abstract domains. 

The most important ideas are as follows: 

• A game can be defined by the initial state (how the board is set up), the operators (which 
define the legal moves), a terminal test (which says when the game is over), and a utility 
or payoff function (which says who won, and by how much). 

• In two-player games with perfect information, the minimax algorithm can determine the 
best move for a player (assuming the opponent plays perfectly) by enumerating the entire 
game tree. 

• The alpha-beta algorithm does the same calculation as minimax, but is more efficient 
because it prunes away branches of the search tree that it can prove are irrelevant to the 
final outcome. 

• Usually, it is not feasible to consider the whole game tree (even with alpha-beta), so we 
need to cut off the search at some point and apply an evaluation function that gives an 
estimate of the utility of a state. 

• Games of chance can be handled by an extension to the minimax algorithm that evaluates 
chance nodes by taking the average utility of all its children nodes, weighted by the 
probability of the child. 



BIBLIOGRAPHICAL AND HISTORICAL NOTES 

The early history of mechanical game playing was marred by numerous frauds. The most 
notorious of these was Baron Wolfgang von Kempelen's "Turk," exhibited in 1769, a supposed 
chess-playing automaton whose cabinet actually concealed a diminutive human chess expert 
during play. The Turk is described in Harkness and Battell (1947). In 1846, Charles Babbage 
appears to have contributed the first serious discussion of the feasibility of computer game playing 
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(Morrison and Morrison, 1961). He believed that if his most ambitious design for a mechanical 
digital computer, the Analytical Engine, were ever completed, it could be programmed to play 
checkers and chess. He also designed, but did not build, a special-purpose machine for playing 
Tic-Tac-Toe. Ernst Zermelo, the designer of modem axiomatic set theory, later speculated on 
the rather quixotic possibility of searching the entire game tree for chess in order to determine 
a perfect strategy (Zermelo, 1976). The first functioning (and nonfraudulent) game-playing 
machine was designed and built around 1 890by the Spanish engineer Leonardo Torres y Quevedo. 
It specialized in the “KRK” chess endgame (king and rook vs. king), playing the side with the 
king and rook against a human opponent attempting to defend with the lone king. Its play was 
correct and it was capable of forcing mate from any starting position (with the machine moving 
first). The “Nimotron” (Condon et al„ 1940) demonstrated perfect play for the very simple 
game ofNim. Significantly, a completely optimal strategy forNim and an adequate strategy for 
the KRK chess endgame (i.e., one which will always win when given the first move, although 
not necessarily in the minimal number of moves) are both simple enough to be memorized and 
executed algorithmically by humans. 

Torres y Quevedo's achievement, and even Babbage's and Zermelo's speculations, re- 
mained relatively isolated until the mid- 1940s — the era when programmable electronic digital 
computers were first being developed. The comprehensive theoretical analysis of game strategy 
in Theory of Games and Economic Behavior (Von Neumann and Morgenstem, 1944) placed 
emphasis on minimaxing (without any depth cutoff) as a way to define mathematically the 
game-theoretic value of a position in a game. Konrad Zuse (1945), the first person to design 
a programmable computer, developed ideas as to how mechanical chess play might be accom- 
plished. Adriaan de Groot (1946) carried out in-depth psychological analysis of human chess 
strategy, which was useful to designers of computer chess programs. Norbert Wiener's (1948) 
book Cybernetics included a brief sketch of the functioning of a possible computer chess-playing 
program, including the idea of using minimax search with a depth cutoff and an evaluation func- 
tion to select a move. Claude Shannon (1950) wrote a highly influential article that laid out the 
basic principles underlying modern computer game-playing programs, although the article did 
not actually include a program of his own. Shannon described minimaxing with a depth cutoff 
and evaluation function more clearly and in more detail than had Wiener, and introduced the 
notion of quiescence of a position. Shannon also described the possibility of using nonexhaustive 
("type B”) as opposed to exhaustive ("type A") minimaxing. Slater (1950) and the commentators 
on his article in the same volume also explored the possibilities for computer chess play. In 
particular, Good ( 1 950) developed the notion of quiescence independently of Shannon. 

In 1951, Alan Turing wrote the first actual computer program capable of playing a full 
game of chess. (The program was published in Turing (1953).) But Turing's program never 
actually ran on a computer; it was tested by hand simulation against a very weak human player, 
who defeated it. Meanwhile D. G. Prinz (1952) had written, and actually run, a program that 
solved chess problems, although it did not play a full game. 

Checkers, rather than chess, was the first of the classic games for which a program actually 
running on a computer was capable of playing out a full game. Christopher Strachey (1952) was 
the first to publish such research, although Slagle (1971) mentions a checkers program written 
by Arthur Samuel as early as 1947. Chinook, the checkers program that recently took over the 
world title from Marion Tinsley, is described by Schaefferet al. (1992). 
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A group working at Los Alamos (Kister et ai, 1957) designed and ran a program that 
played a full game of a variant of chess using a 6 x 6 board. Alex Bernstein wrote the first 
program to play a full game of standard chess (Bernstein and Roberts, 1958; Bernstein et ai, 
1958), unless possibly this feat was accomplished by the Russian BESM program mentioned in 
Newell etal. (1958), about which little information is available. 

John McCarthy conceived the idea of alpha-beta search in 1 956, although he did not publish 
it. The NSS chess program (Newell et al., 1958) used a simplified version of alpha-beta; it was 
the first chess program to do so. According to Nilsson ( 1971), Arthur Samuel's checkers program 
(Samuel, 1959; Samuel, 1967) also used alpha-beta, although Samuel did not mention it in the 
published reports on the system. Papers describing alpha-beta were published in the early 1960s 
(Hart and Edwards, 1961; Brudno, 1963; Slagle, 1963b). An implementation of full alpha-beta 
is described by Slagle and Dixon (1969) in a program for playing the game of kalah. Alpha-beta 
was also used by the “Kotok-McCarthy” chess program written by a student of John McCarthy 
(Kotok, 1962) and by the MacHack 6 chess program (Greenblatt et al., 1967). MacHack 6 was 
the first chess program to compete successfully with humans, although it fell considerably short 
of Herb Simon's prediction in 1957 that a computer program would be world chess champion 
within 10 years (Simon and Newell, 1958). Knuth and Moore (1975) provide a history of alpha- 
beta, along with a proof of its correctness and a time complexity analysis. Further analysis of 
the effective branching factor and time complexity of alpha-beta is given by Pearl (1982b). Pearl 
shows alpha-beta to be asymptotically optimal among all game-searching algorithms. 

It would be a mistake to infer that alpha-beta's asymptotic optimality has completely 
suppressed interest in other game-searching algorithms. The best-known alternatives are probably 
the B* algorithm (Berliner, 1979), which attempts to maintain interval bounds on the possible 
value of a node in the game tree, rather than giving it a single point-valued estimate as minimax 
and alpha-beta do, and SSS* (Stockman, 1979), which dominates alpha-beta in the sense that 
the set of nodes in the tree that it examines is a (sometimes proper) subset of those examined by 
alpha-beta. Palay (1985) uses probability distributions in place of the point values of alpha-beta 
or the intervals of B*. David McAllester's (1988) conspiracy number search is an interesting 
generalization of alpha-beta. MGSS* (Russell and Wefald, 1989) uses the advanced decision- 
theoretic techniques of Chapter 16 to decide which nodes to examine next, and was able to 
outplay an alpha-beta algorithm at Othello despite searching an order of magnitude fewer nodes. 
Individual games are subject to ad hoc mathematical analysis; a fascinating study of a huge 
number of games is given by Berlekamp et al. (1982). 

D. F. Beal (1980) and DanaNau (1980; 1983) independently and simultaneously showed 
that under certain assumptions about the game being analyzed, any form of minimaxing, including 
alpha-beta, using an evaluation function, yields estimates that are actually less reliable than the 
direct use of the evaluation function, without any search at all! Heuristics (Pearl, 1984) gives 
a thorough analysis of alpha-beta and describes B*. SSS*, and other alternative game search 
algorithms. It also explores the reasons for the Beal/Nau paradox, and why it does not apply 
to chess and other games commonly approached via automated game-tree search. Pearl also 
describes AND/OR graphs (Slagle, 1963a), which generalize game-tree search but can be applied 
to other types of problems as well, and the AO* algorithm (Martelli and Montanari, 1973; 
Martelli and Montanari, 1978) for searching them. Kaindl (1990) gives another survey of 
sophisticated search algorithms. 
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The first two computer chess programs to play a match against each other were the Kotok- 
McCarthy program and the “ITEP” program written at Moscow's Institute of Theoretical and 
Experimental Physics (Adelson- Velsky et al., 1970). This intercontinental match was played by 
telegraph. It ended in 1967 with a 3-1 victory for the ITEP program. The first ACM North 
American Computer Chess Championship tournament was held in New York City in 1970. The 
first World Computer Chess Championship was held in 1974 in Stockholm (Hayes and Levy, 
1976). It was won by Kaissa (Adelson-Velsky et al., 1975), another program from ITEP. 

A later version of Greenblatt's MacHack 6 was the first chess program to run on custom 
hardware designed specifically for chess (Moussouris et al., 1 979), but the first program to achieve 
notable success through the use of custom hardware was Belle (Condon and Thompson, 1982). 
Most of the strongest recent programs, such as HITECH (Ebeling, 1987; Berliner and Ebeling, 
1989) and Deep Thought (Hsu et al., 1990) have run on custom hardware. Major exceptions 
are Cray Blitz (Hyatt et al., 1986), which runs on a general-purpose Cray supercomputer, and 
Socrates II, winner of the 23rd ACM North American Computer Chess Championship in 1993, 
which runs on an Intel 486-based microcomputer. It should be noted that Deep Thought was not 
there to defend its title. Deep Thought 2 regained the championship in 1994. It should also be 
noted that even custom-hardware machines can benefit greatly from improvements purely at the 
software level (Berliner, 1989). 

The Fredkin Prize, established in 1980, offered $5000 to the first program to achieve a 
Master rating, $10,000 to the first program to achieve a USCF (United States Chess Federation) 
rating of 2500 (near the grandmaster level), and $100,000 for the first program to defeat the 
human world champion. The $5000 prize was claimed by Belle in 1983, and the $10,000 prize 
by Deep Thought in 1989. The $100,000 prize remains unclaimed, in view of convincing wins 
in extended play by world champion Gary Kasparov over Deep Thought (Hsu et al., 1990). 

The literature for computer chess is far better developed than for any other game played 
by computer programs. Aside from the tournaments already mentioned, the rather misleadingly 
named conference proceedings Heuristic Programming in Artificial Intelligence report on the 
Computer Chess Olympiads. The International Computer Chess Association (ICCA), founded 
in 1977, publishes the quarterly ICCA Journal. Important papers have been published in the 
numbered serial anthology Advances in Computer Chess, starting with (Clarke, 1977). Some 
early general Al textbooks (Nilsson, 1971; Slagle, 1971) include extensive material on game- 
playing programs, including chess programs. David Levy’s Computer Chess Compendium (Levy, 
1988a) anthologizes many of the most important historical papers in the field, together with the 
scores of important games played by computer programs. The edited volume by Marsland and 
Schaeffer (1990) contains interesting historical and theoretical papers on chess and Go along 
with descriptions of Cray Blitz, HITECH, and Deep Thought. Several important papers on chess, 
along with material on almost all games for which computer game-playing programs have been 
written (including checkers, backgammon, Go, Othello, and several card games) can be found in 
Levy (1988b). There is even a textbook on how to write a computer game-playing program, by 
one of the major figures in computer chess (Levy, 1983). 

The expectimax algorithm described in the text was proposed by Donald Michie (1966), 
although of course it follows directly from the principles of game-tree evaluation due to Von 
Neumann and Morgenstern. Bruce Ballard (1983) extended alpha-beta pruning to cover trees 
with chance nodes. The backgammon program BKG (Berliner, 1977; Berliner, 1980b) was 
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the first program to defeat a human world champion at a major classic game (Berliner, 1980a), 
although Berliner was the first to acknowledge that this was a very short exhibition match (not a 
world championship match) and that BKG was very lucky with the dice. 

The first Go-playing programs were developed somewhat later than those for checkers and 
chess (Lefkovitz, 1960; Remus, 1962) and have progressed more slowly. Ryder (1971) used a 
search-based approach similar to that taken by most chess programs but with more selectivity to 
overcome the enormous branching factor. Zobrist (1970) used a pattern-recognition approach. 
Reitman and Wilcox (1979) used condition-action rules based on complex patterns, combined 
with highly selective localized search. The Go Explorer and its successors (Kierulfet al.. 1990) 
continue to evolve along these lines. YUGO (Shirayanagi, 1990) places heavy emphasis on 
knowledge representation and pattern knowledge. The Computer Go Newsletter, published by 
the Computer Go Association, describes current developments. 



EXERCISES 

5.1 This problem exercises the basic concepts of game-playing using Tic-Tac-Toe (noughts and 
crosses) as an example. We define X n as the number of rows, columns, or diagonals with exactly 
n X’s and no O’s. Similarly, O n is the number of rows, columns, or diagonals withjust n O's. The 
utility function thus assigns +1 to any position with A 3 = 1 and —1 to any position with O 3 - 1 . 
All other terminal positions have utility 0. We will use a linear evaluation function defined as 
Eval = 3X 2 +X l -( 30 2 + 0i) 

a. Approximately how many possible games of Tic-Tac-Toe are there? 

b. Show the whole game tree starting from an empty board down to depth 2, (i.e., one X and 
one O on the board), taking symmetry into account. You should have 3 positions at level 
1 and 12 at level 2 . 

c. Mark on your tree the evaluations of all the positions at level 2. 

d. Mark on your tree the backed-up values for the positions at levels 1 and 0, using the 
minimax algorithm, and use them to choose the best starting move. 

e. Circle the nodes at level 2 that would not be evaluated if alpha-beta pruning were applied, 
assuming the nodes are generated in the optimal order for alpha-beta pruning. 

- 5.2 Implement a general game-playing agent for two-player deterministic games, using alpha- 

beta search. You can assume the game is accessible, so the input to the agent is a complete 
description of the state. 

5.3 Implement move generators and evaluation functions for one or more of the following 
- games: kalah, Othello, checkers, chess. Exercise your game-playing agent using the implemen- 

tation. Compare the effect of increasing search depth, improving move ordering, and improving 
the evaluation function. How close does your effective branching factor come to the ideal case 
of perfect move ordering? 
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5.4 The algorithms described in this chapter construct a search tree for each move from scratch. 
Discuss the advantages and disadvantages of retaining the search tree from one move to the next 
and extending the appropriate portion. How would tree retention interact with the use of selective 
search to examine "useful" branches of the tree? 

5.5 Develop a formal proof of correctness of alpha-beta pruning. To do this, consider the 
situation shown in Figure 5.14. The question is whether to prune node rij, which is a max-node 
and a descendant of node n\ . The basic idea is to prune it if and only if the minimax value of n\ 
can be shown to be independent of the value of rij. 

a. The value of n\ is given by 

«i =min(n 2 ,M2i,...,M 2 6,) 

By writing a similar expression for the value of u 2 . find an expression for n\ in terms of rij. 

b. Let It be the minimum (or maximum) of the node values to the left of node rij at depth i. 
These are the nodes whose minimax value is already known. Similarly, let r, be the 
minimum (or maximum) of the node values to the right of rij at depth i. These nodes have 
not yet been explored. Rewrite your expression for n\ in terms of the /, and r, values. 

c. Now reformulate the expression to show that in order to affect n\ , rij must not exceed a 
certain bound derived from the l, values. 

d. Repeat the process for the case where rij is a min-node. 

You might want to consult Wand (1980), who shows how the alpha-beta algorithm can be auto- 
matically synthesized from the minimax algorithm, using some general program-transformation 
techniques. 
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5.6 Prove that with a positive linear transformation of leaf values, the move choice remains 
unchanged in a game tree with chance nodes. 

5.7 Consider the following procedure for choosing moves in games with chance nodes: 

• Generate a suitable number (say, 50) dice-roll sequences down to a suitable depth (say, 8). 

• With known dice rolls, the game tree becomes deterministic. For each dice-roll sequence, 
solve the resulting deterministic game tree using alpha-beta. 

• Use the results to estimate the value of each move and choose the best. 

Will this procedure work correctly? Why (not)? 

5.8 Let us consider the problem of search in a three-player game. (You can assume no alliances 
are allowed for now.) We will call the players 0, 1, and 2 for convenience. The first change is 
that the evaluation function will return a list of three values, indicating (say) the likelihood of 
winning for players 0, 1, and 2, respectively. 

a. Complete the following game tree by filling in the backed-up value triples for all remaining 
nodes, including the root: 




b. Rewrite Minimax-Decision and Minimax- Value so that they work correctly for the 
three-player game. 

c. Discuss the problems that might arise if players could form and terminate alliances as well 
as make moves "on the board." Indicate briefly how these problems might be addressed. 

5.9 Describe and implement a general game-playing environment for an arbitrary number of 
players. Remember that time is part of the environment state, as well as the board position. 

5.10 Suppose we play a variant of Tic-Tac-Toe in which each player sees only his or her own 
moves. If the player makes a move on a square occupied by an opponent, the board "beeps" and 
the player gets another try. Would the backgammon model suffice for this game, or would we 
need something more sophisticated? Why? 
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5.11 Describe and/or implement state descriptions, move generators, and evaluation functions 
for one or more of the following games: backgammon. Monopoly, Scrabble, bridge (declarer 
play is easiest). 

5.12 Consider carefully the interplay of chance events and partial information in each of the 
games in Exercise 5.11. 

a. For which is the standard expectiminimax model appropriate? Implement the algorithm 
and run it in your game-playing agent, with appropriate modifications to the game-playing 
environment. 

b. For which would the scheme described in Exercise 5.7 be appropriate? 

c. Discuss how you might deal with the fact that in some of the games, the players do not 
have the same knowledge of the current state. 

5.13 The Chinook checkers program makes extensive use of endgame databases, which provide 
exact values for every position within 6 moves of the end of the game. How might such databases 
be generated efficiently? 

5.14 Discuss how well the standard approach to game playing would apply to games such as 
tennis, pool, and croquet, which take place in a continuous, physical state space. 

5.15 For a game with which you are familiar, describe how an agent could be defined with 
condition-action rules, subgoals (and their conditions for generation), and action-utility rules, 
instead of by minimax search. 

5.16 The minimax algorithm returns the best move for MAX under the assumption that MIN 
plays optimally. What happens when MIN plays suboptimally? 

5.17 We have assumed that the rules of each game define a utility function that is used by both 
players, and that a utility ofx for MAX means a utility of — x for MIN. Games with this property 
are called zero-sum games. Describe how the minimax and alpha-beta algorithms change when 
we have nonzero-sum games — that is, when each player has his or her own utility function. You 
may assume that each player knows the other's utility function. 



Part III 



KNOWLEDGE AND REASONING 



In Part II, we showed that an agent that has goals and searches for solutions to 
the goals can do better than one that just reacts to its environment. We focused 
mainly on the question of how to carry out the search, leaving aside the question 
of general methods for describing states and actions. 

In this part, we extend the capabilities of our agents by endowing them with 
the capacity for general logical reasoning. A logical, knowledge-based agent 
begins with some knowledge of the world and of its own actions. It uses logical 
reasoning to maintain a description of the world as new percepts arrive, and to 
deduce a course of action that will achieve its goals. 

In Chapter 6, we introduce the basic design for a knowledge -based agent. 
We then present a simple logical language for expressing knowledge, and show 
how it can be used to draw conclusions about the world and to decide what to 
do. In Chapter 7, we augment the language to make it capable of expressing a 
wide variety of knowledge about complex worlds. In Chapter 8, we exercise this 
capability by expressing a significant fragment of commonsense knowledge about 
the real world, including time, change, objects, categories, events, substances, 
and of course money. In Chapters 9 and 10 we discuss the theory and practice of 
computer systems for logical reasoning. 
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A Knowledge-Based AGENT 



KNOWLEDGEBASE The central component of a knowledge-based agent isits knowledge base, or KB. Informally, a 
knowledgebase is a set of representations of facts about the world. Each individual representation 
SENTENCE is called a sentence. (Here "sentence" is used as a technical term. It is related to the sentences 
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of English and other natural languages, but is not identical.) The sentences are expressed in a 
representation language called a knowledge representation language. 

There must be a way to add new sentences to the knowledge base, and a way to query what 
is known. The standard names for these tasks are TELL and Ask, respectively. The fundamental 
requirement that we will impose on TELL and ASK is that when one ASKS a question of the 
knowledge base, the answer should follow from what has been told (or rather, TELLed) to the 
knowledge base previously. Later in the chapter, we will be more precise about the crucial word 
"follow." For now, take it to mean that the knowledge base should notjust make up things as 
it goes along. Determining what follows from what the KB has been TELLed is the job of the 
inference inference mechanism, the other main component of a knowledge-based agent. 

Figure 6. 1 shows the outline of a knowledge-based agent program. Like all our agents, it 
takes a percept as input and returns an action. The agent maintains a knowledge base, KB, which 
knowledge 0 may initially contain some background knowledge. Each time the agent program is called, it 
does two things. First, it TELLS the knowledge base what it perceives. 1 Second, it ASKS the 
knowledge base what action it should perform. In the process of answering this query, logical 
reasoning is used to prove which action is better than all others, given what the agent knows and 
what its goals are. The agent then performs the chosen action. 



function KB- Agent (percept ) returns an action 
static: KB, a knowledge base 

t, a counter, initially 0, indicating time 

TELL(KB, MAKE-PERCEPT-SENTENCE(percept, t)) 
action — Ask(KB. M ake-Agtion-Query( t)) 
TELL(A'B, MAKE-ACTION-SENTENCE(0C?iO«, ?)) 
t<—t + 1 
return action 



Figure 6.1 A generic knowledge-based agent. 



The details of the representation language are hidden inside two functions that implement 
the interface between the agent program "shell" and the core representation and reasoning system. 
MAKE-PERCEPT-SENTENCE takes a percept and a time and returns a sentence representing the fact 
that the agent perceived the percept at the given time, and Make-Action-Query takes a time as 
input and returns a sentence that is suitable for asking what action should be performed at that 
time. The details of the inference mechanism are hidden inside TELL and ASK. Later sections 
will reveal these details. 

A careful examination of Figure 6.1 reveals that it is quite similar to the design of agents 
with internal state described in Chapter 2. Because of the definitions of TELL and ASK, however, 
the k nowledge-based agent is not an arbitrary program for calculating actions based on the internal 
state variable. At any point, we can describe a knowledge-based agent at three levels: 

1 You might think of TELL and ASK as procedures that humans can use to communicate with knowledge bases. Don't 
be confused by the fact that here it is the agent that is TELLing things to its own knowledge base. 
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The knowledge level or epistemological level is the most abstract; we can describe the 
agent by saying what it knows. For example, an automated taxi might be said to know that 
the Golden Gate Bridge links San Francisco and Marin County. If TELL and ASK work 
correctly, then most of the time we can work at the knowledge level and not worry about 
lower levels. 

The logical level is the level at which the knowledge is encoded into sentences. For example, 
the taxi might be described as having the logical sentence Links(GGBridge,SF, Marin ) in 
its knowledge base. 

The implementation level is the level that runs on the agent architecture. It is the 
level at which there are physical representations of the sentences at the logical level. A 
sentence such as Links{GGBridge,SF, Marin) could be represented in the KB by the string 
" Links (GGBridge, SF, Marin) " contained in a list of strings; or by a "1" entry in 
a three-dimensional table indexed by road links and location pairs; or by a complex set 
of pointers connecting machine addresses corresponding to the individual symbols. The 
choice of implementation is very important to the efficient performance of the agent, but it 
is irrelevant to the logical level and the knowledge level. 

We said that it is possible to understand the operation of a knowledge-based agent in terms of 
. • what it knows. It is possible to construct a knowledge-based agent by JELLing it what it needs 

• to know. The agent's initial program, before it starts to receive percepts, is built by adding one 

by one the sentences that represent the designer's knowledge of the environment. Provided that 
the representation language makes it easy to express this knowledge in the form of sentences, 
declarative this simplifies the construction problem enormously. This is called the declarative approach 

to system building. Also, one can design learning mechanisms that output general knowledge 
about the environment given a series of percepts. By hooking up a learning mechanism to a 
knowledge-based agent, one can make the agent fully autonomous. 
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6,2 THE WUMPUS WORLD ENVIRONMENT 



Before launching into a full exposition of knowledge representation and reasoning, we will 
WUMPUS WORLD describe a simple environment class — the wumpus world — that provides plenty of motivation 
for logical reasoning. Wumpus was an early computer game, based on an agent who explores 
a cave consisting of rooms connected by passageways. Lurking somewhere in the cave is the 
wumpus, a beast that eats anyone who enters its room. To make matters worse, some rooms 
contain bottomless pits that will trap anyone who wanders into these rooms (except for the 
wumpus, who is too big to fall in). The only mitigating feature of living in this environment is 
the occasional heap of gold. 

It turns out that the wumpus game is rather tame by modern computer game standards. 
However, it makes an excellent testbed environment for intelligent agents. Michael Genesereth 
was the first to suggest this. 
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Specifying the environment 

Like the vacuum world, the wumpus world is a grid of squares surrounded by walls, where each 
square can contain agents and objects. The agent always starts in the lower left comer, a square 
that we will label [1,1]. The agent's task is to find the gold, return to [1,1] and climb out of the 
cave. An example wumpus world is, shown in Figure 6.2. 




Figure 6.2 A typical wumpus world. 



To specify the agent's task, we specify its percepts, actions, and goals. In the wumpus 
world, these are as follows: 

• In the square containing the wumpus and in the directly (not diagonally) adjacent squares 
the agent will perceive a stench. 

• In the squares directly adjacent to a pit, the agent will perceive a breeze. 

• In the square where the gold is, the agent will perceive a glitter. 

• When an agent walks into a wall, it will perceive a bump. 

• When the wumpus is killed, it gives out a woeful scream that can be perceived anywhere 
in the cave. 

• The percepts will be given to the agent in the form of a list of five symbols; for example, 
if there is a stench, a breeze, and a glitter but no bump and no scream, the agent will 
receive the percept [Stench, Breeze, Glitter, None, None], The agent cannot perceive its 
own location. 

• Just as in the vacuum world, there are actions to go forward, turn right by 90° , and turn left 
by 90° . In addition, the action Grab can be used to pick up an object that is in the same 
square as the agent. The action Shoot can be used to fire an arrow in a straight line in the 
direction the agent is facing. The arrow continues until it either hits and kills the wumpus 
or hits the wall. The agent only has one arrow, so only the first Shoot action has any effect. 
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Finally, the action Climb is used to leave the cave; it is effective only when the agent is in 
the start square. 

• The agent dies a miserable death if it enters a square containing a pit or a live wumpus. It 
is safe (but smelly) to enter a square with a dead wumpus. 

• The agent's goal is to find the gold and bring it back to the start as quickly as possible, 
without getting killed. To be precise, 1000 points are awarded for climbing out of the 
cave while carrying the gold, but there is a 1 -point penalty for each action taken, and a 
10,000-pointpenalty for getting killed. 

As we emphasized in Chapter 2, an agent can do well in a single environment merely by 
memorizing the sequence of actions that happens to work in that environment. To provide a real 
test, we need to specify a complete class of environments, and insist that the agent do well, on 
average, over the whole class. We will assume a 4 x 4 grid surrounded by walls. The agent 
always starts in the square labeled (1,1), facing toward the right. The locations of the gold and 
the wumpus are chosen randomly, with a uniform distribution, from the squares other than the 
start square. In addition, each square other than the start can be a pit, with probability 0.2. 

In most of the environments in this class, there is a way for the agent to safely retrieve the 
gold. In some environments, the agent must choose between going home empty-handed or taking 
a chance that could lead either to death or to the gold. And in about 21% of the environments 
(the ones where the gold is in a pit or surrounded by pits), there is no way the agent can get a 
positive score. Sometimes life is just unfair. 

After gaining experience with this class of environments, we can experiment with other 
classes. In Chapter 22 we consider worlds where two agents explore together and can commu- 
nicate with each other. We could also consider worlds where the wumpus can move, or where 
there are multiple troves of gold, or multiple wumpuses. 2 



Acting and reasoning in the wumpus world 

We now know the rules of the wumpus world, but we do not yet have an idea of how a wumpus 
world agent should act. An example will clear this up and will show why a successful agent 
will need to have some kind of logical reasoning ability. Figure 6.3(a) shows an agent's state of 
knowledge at the start of an exploration of the cave in Figure 6.2, after it has received its initial 
percept. To emphasize that this is only a representation, we use letters such as A and OK to 
represent sentences, in contrast to Figure 6.2, which used (admittedly primitive) pictures of the 
wumpus and pits. 

From the fact that there was no stench or breeze in [1,1], the agent can infer that [1,2] and 

[2.1] are free of dangers. They are marked with an OK to indicate this. From the fact that the 
agent is still alive, it can infer that [1,1 ] is also OK. A cautious agent will only move into a square 
that it knows is OK. Let us suppose the agent decides to move forward to [2,1], giving the scene 
in Figure 6.3(b). 

The agent detects a breeze in [2,1], so there must be a pit in a neighboring square, either 

[2.2] or [3,1]. The notation P? indicates a possible pit. The pit cannot be in [1,1], because the 



Or is it wumpi? 



Figure 6.3 The first step taken by the agent in the wumpus world, (a) The initial sit- 
uation, after percept [None, None, None, None, None], (b) After one move, with percept 

[None, Breeze, None, None, None]. 




Figure 6.4 Two later stages in the progress of the agent, (a) After the third move, 
with percept [Stench. None. None. None. None], (b) After the fifth move, with percept 
[Stench, Breeze, Glitter, None, None], 



agent was already there and did not fall in. At this point, there is only one known square that is 
OK and has not been visited yet. So the prudent agent will turn around, go back to [1,1], and 
then proceed to [1,2], giving the state of knowledge in Figure 6.4(a). 

The agent detects a stench in [1,2], which means that there must be a wumpus nearby. But 
the wumpus cannot be in [1,1] (or it would have eaten the agent at the start), and it cannot be in 
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[2,2] (or the agent would have detected a stench when it was in [2,1]). Therefore, the agent can 
infer that the wumpus is in [1,3]. The notation W \ indicates this. More interesting is that the 
lack of a Breeze percept in [1,2] means that there must be a pit in [3,1]. The reasoning is that no 
breeze in [1,2] means there can be no pit in [2,2], But we already inferred that there must be a 
pit in either [2,2] or [3,1], so this means it must be in [3,1]. This is a fairly difficult inference, 
because it combines knowledge gained at different times in different places, and relies on the 
lack of a percept to make one crucial step. The inference is beyond the abilities of most animals, 
but it is typical of the kind of reasoning that a logical agent does. 

After these impressive deductions, there is only one known unvisited OK square left, [2,2], 
so the agent will move there. We will not show the agent's state ofknowledge at [2,2]; we just 
assume the agent turns and moves to [2,3], giving us Figure 6.4(b). In [2,3], the agent detects a 
glitter, so it should grab the gold and head for home, making sure its return trip only goes through 
squares that are known to be OK. 

In the rest of this chapter, we describe how to build a logical agent that can represent beliefs 
such as "there is a pit in [2,2] or [3,1]” and "there is no wumpus in [2,2],” and that can make all 
the inferences that were described in the preceding paragraphs. 



6.3 REPRESENTATION. REASONING. AND LOGIC 



In this section, we will discuss the nature of representation languages, and of logical languages 
in particular, and explain in detail the connection between the language and the reasoning 
mechanism that goes with it. Together, representation and reasoning support the operation of a 
knowledge-based agent. 

represewation The object of knowledge representation is to express knowledge in computer-tractable 

form, such that it can be used to help agents perform well. A knowledge representation language 
is defined by two aspects: 

syntax • The syntax of a language describes the possible configurations that can constitute sentences. 

Usually, we describe syntax in terms of how sentences are represented on the printed page, 
but the real representation is inside the computer: each sentence is implemented by a 
physical configuration or physical property of some part of the agent. For now, think of 
this as being a physical pattern of electrons in the computer's memory. 

semantics • The semantics determines the facts in the world to which the sentences refer. Without 

semantics, a sentence is just an arrangement of electrons or a collection of marks on a page. 
With semantics, each sentence makes a claim about the world. And with semantics, we 
can say that when a particular configuration exists within an agent, the agent believes the 
corresponding sentence. 

For example, the syntax of the language of arithmetic expressions says that if x and y are 
expressions denoting numbers, then x > y is a sentence about numbers. The semantics of the 
language says that x > y is false when y is a bigger number than x, and true otherwise. 
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Provided the syntax and semantics are defined precisely, we can call the language a logic . 3 
From the syntax and semantics, we can derive an inference mechanism for an agent that uses the 
language. We now explain how this comes about. 

First, recall that the semantics of the language determine the fact to which a given sentence 
refers (see Figure 6.5). It is impprtant to distinguish between facts and their representations. 
Facts are part of the world, 4 whereas their representations must be encoded in some way that can 
be physically stored within an agent. We cannot put the world inside a computer (nor can we put 
it inside a human), so all reasoning mechanisms must operate on representations of facts, rather 
than on the facts themselves. Because sentences are physical configurations of parts of the agent, 
reasoning must be a process of constructing new physical configurations from old ones. Proper 
reasoning should ensure that the new configurations represent facts that actually follow from the 
facts that the old configurations represent. 
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Figure 6.5 The connection between sentences and facts is provided by the semantics of the 
language. The property of one fact following from some other facts is mirrored by the property of 
one sentence being entailed by some other sentences. Logical inference generates new sentences 
that are entailed by existing sentences. 



Consider the following example. From the fact that the solar system obeys the laws of 
gravitation, and the fact of the current arrangement of the sun, planets, and other bodies, it follows 
(so the astronomers tell us) thatPluto will eventually spin off into the interstellar void. But if our 
agent reasons improperly, it might start with representations of the first two facts and end with a 
representation that means that Pluto will shortly arrive in the vicinity of Bucharest. Or we might 
end up with "logical" reasoning like that in Figure 6.6. 

We want to generate new sentences that are necessarily true, given that the old sentences 
entailment are true. This relation between sentences is called entailment, and mirrors the relation of one 

fact following from another (Figure 6.5). In mathematical notation, the relation of entailment 
between a knowledge base KB and a sentence a is pronounced "Oentails a” and written as 
KB l=o. ‘"'j ‘ . 

An inference procedure can do one of two things: given a knowledge base KB, it can generate 

3 This is perhaps a rather broad interpretation of the term "logic," one that makes "representation language" and "logic" 
synonymous. However, most of the principles of logic apply at this general level, rather than just at the level of the 
particular languages most often associated with the term. 

4 As Wittgenstein (1922) put it in his famous Tractatus Logico-Philosophicus: "The world is everything that is the 
case." We are using the word "fact" in this sense: as an "arrangement" ofthe world that may or may not be the case. 
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SOUND 

TRUTH-PRESERVING 



FIRST VILLAGER: We have found a witch. May we bum her? 

ALL: A witch! Burnher! 

BEDEVERE: Why do you think she is a witch? 

SECOND VILLAGER: She turned me into a newt. 

BEDEVERE: A newt? 

SECOND VILLAGER (after looking at himself for some time): I got better. 

ALL: Burn her anyway. 

BEDEVERE: Quiet! Quiet! There are ways of telling whether she is a witch. 
BEDEVERE: Tell me . . . what do you do with witches? 

ALL: Burn them. 

BEDEVERE: And what do you burn, apart from witches? 

FOURTH VILLAGER: ...Wood? 

BEDEVERE: So why do witches bum? 

SECOND VILLAGER: (pianissimo) Because they're made of wood? 

BEDEVERE: Good. 

ALL: I see. Yes, of course. 

BEDEVERE: So how can we tell if she is made of wood? 

FIRST VILLAGER: Make abridge out of her. 

BEDEVERE: Ah ... but can you not also make bridges out of stone? 

ALL: Yes, of course . . . um . . . er . . . 

BEDEVERE: Does wood sink in water? 

ALL: No, no, it floats. Throw her in the pond. 

BEDEVERE: Wait. Wait . . . tell me, what also floats on water? 

ALL: Bread? No, no no. Apples . . . gravy . . . very small rocks . . . 

BEDEVERE: No, no no, 

King ARTHUR: A duck! 

(They all turn and lookat ARTHUR. BEDEVERE looks up very impressed.) 
BEDEVERE: Exactly. So ... logically . . . 

FIRST VILLAGER (beginning to pick up the thread): If she ... weighs the same as 
a duck . . . she's made of wood. 

BEDEVERE: And therefore? 

ALL: A witch! 



Figure 6.6 An example of "logical" reasoning gone wrong. (Excerpted with permission from 
Monty Python and the Holy Grail, © 1 977, Reed Consumer Books.) 



new sentences a that purport to be entailed by KB. Or, given a knowledge base KB and another 
sentence a, it can report whether or not a is entailed by KB. An inference procedure that generates 
only entailed sentences is called sound or truth-preserving. 

An inference procedure i can be described by the sentences that it can derive. If i can 
derive a from KB, a logician would write 
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which is pronounced "Alpha is derived from KB by i” or “/ derives alpha from KB'' Sometimes 
the inference procedure is implicit and the i is omitted. The record of operation of a sound 
proof inference procedure is called a proof. 

In understanding entailment and proof, it may help to think of the set of all consequences 
of KB as a haystack and a as a needle. Entailment is like the needle being in the haystack; proof 
is like finding it. For real haystacks, which are finite in extent, it seems obvious that a systematic 
examination can always decide whether the needle is in the haystack. This is the question 
complete of completeness: an inference procedure is complete if it can find a proof for any sentence 

that is entailed. But for many knowledge bases, the haystack of consequences is infinite, and 
completeness becomes an important issue. 5 

■ We have said that sound inference is desirable. How is it achieved? The key to sound 

I 1 ' inference is to have the inference steps respect the semantics of the sentences they operate upon. 

That is, given a knowledge base, KB, the inference steps should only derive new sentences that 
represent facts that follow from the facts represented by KB. By examining the semantics of 
proof theory logical languages, we can extract what is called the proof theory of the language, which specifies 
the reasoning steps that are sound. Consider the following familiar example from mathematics, 
which illustrates syntax, semantics, and proof theory. Suppose we have the following sentence: 
E = me 2 

The syntax of the "equation language" allows two expressions to be connected by an “=” sign. 
An expression can be a simple symbol or number, a concatenation of two expressions, two 
expressions joined by a “+” sign, and so on. The semantics of the language says that the 
two expressions on each side of “=” refer to the same quantity; that the concatenation of two 
expressions refers to the quantity that is the product of the quantities referred to by each of the 
expressions; and so on. From the semantics, we can show that a new sentence can be generated 
by, for example, concatenating the same expression to both sides of the equation: 

ET=mc 2 T 

Most readers will have plenty of experience with inference of this sort. Logical languages are like 
this simple equation language, but rather than dealing with algebraic properties and numerical 
quantities, they must deal with more or less everything we might want to represent and about 
which we might want to reason. 

Representation 

We will now look a little more deeply into the nature of knowledge representation languages, 
with the aim of designing an appropriate syntax and semantics. We will begin with two familiar 
classes of languages, programming languages and natural languages, to see what they are good 
at representing and where they have problems. 

Programming languages (such as C or Pascal or Lisp) are good for describing algorithms 
and concrete data structures. We could certainly imagine using an 4 x 4 array to represent 
the contents of the wumpus world, for example. Thus, the programming language statement 
World\2,2 1 — Pit is a fairly natural way to say that there is a pit in square [2,2], However, most 



Compare with the case of infinite search spaces in Chapter 3, where depth-first search is not complete. 
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programming languages do not offer any easy way to say "there is a pit in [2,2] or [3,1]” or 
"there is a wumpus in some square." The problem is that programming languages are designed to 
completely describe the state of the computer and how it changes as the program executes. But 
we would like our knowledge representation language to support the case where we do not have 
complete information — where we do not know for certain how things are, but only know some 
possibilities for how they might or might not be. A language that does not let us do this is not 
expressive enough. 

Natural languages (such as English or Spanish) are certainly expressive — we managed to 
write this whole book using natural language with only occasional lapses into other languages 
(including logic, mathematics, and the language of diagrams). But natural languages have evolved 
more to meet the needs of communication rather than representation. When a speaker points 
and says, "Look!" the listener comes to know that, say, Superman has finally appeared over the 
rooftops. But we would not want to say that the sentence "Look!" encoded that fact. Rather, 
the meaning of the sentence depends both on the sentence itself and on the context in which the 
sentence was spoken. A natural language is a good way for a speaker to get a listener to come 
to know something, but often this sharing of knowledge is done without explicit representation 
of the knowledge itself. Natural languages also suffer from ambiguity — in a phrase such as 
"small dogs and cats," it is not clear whether the cats are small. Contrast this to the programming 
language construct d+ c,” where the precedence rules for the language tell us that the minus 
sign applies to d, not to d + c. 

A good knowledge representation language should combine the advantages of natural 
languages and formal languages. It should be expressive and concise so that we can say everything 
we need to say succinctly. It should be unambiguous and independent of context, so that what we 
say today will still be interpretable tomorrow. And it should be effective in the sense that there 
should be an inference procedure that can make new inferences from sentences in our language. 

Many representation languages have been designed to try to achieve these criteria. In 
this book, we concentrate on first-order logic as our representation language because it forms 
the basis of most representation schemes in AI. Just as it would be overoptimistic to believe 
that one can make real progress in physics without understanding and using equations, it is 
important to develop a talent for working with logical notation if one is to make progress in 
artificial intelligence. However, it is also important not to get too concerned with the specifics 
of logical notation — after all, there are literally dozens of different versions, some with x’s and 
>"s and exotic mathematical symbols, and some with rather visually appealing diagrams with 
arrows and bubbles. The main thing to keep hold of is how a precise, formal language can 
represent knowledge, and how mechanical procedures can operate on expressions in the language 
to perform reasoning. The fundamental concepts remain the same no matter what language is 
being used to represent the knowledge. 



Semantics 

In logic, the meaning of a sentence is what it states about the world, that the world is this way and 
not that way. So how does a sentence get its meaning? How do we establish the correspondence 
between sentences and facts? Essentially, this is up to the person who wrote the sentence. In 
order to say what it means, the writer has to provide an interpretation for it; to say what fact 
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THE LANGUAGE OF THOUGHT 

Philosophers and psychologists have long pondered how it is that humans and other 
animals represent knowledge. It is clear that the evolution of natural language has 
played an important role in developing this ability in humans. But it is also true that 
humans seem to represent much of their knowledge in a nonverbal form. Psychologists 
have done studies to confirm that humans remember the "gist" of something they have 
read rather than the exact words. You could look at Anderson's (1980, page 96) 
description of an experiment by Wanner, or you could perform your own experiment 
by deciding which of the following two phrases formed the opening of Section 6.3: 
"In this section, we will discuss the nature of representation languages . . .” 
"This section covers the topic of knowledge representation languages . . .” 

In Wanner's experiment, subjects made the right choice at chance level — about 50% 
of the time — but remembered the overall idea of what they read with better than 
90% accuracy. This indicates that the exact words are not part of the representations 
they formed. A similar experiment (Sachs, 1967) showed that subjects remember the 
words for a short time (seconds to tens of seconds), but eventually forget the words 
and remember only the meaning. This suggests that people process the words to form 
some kind of nonverbal representation which they maintain as memories. 

The exact mechanism by which language enables and shapes the representation of 
ideas in humans remains a fascinating question. The famous Sapir-Whorfhypothesis 
claims that the language we speak profoundly influences the way in which we think 
and make decisions, in particular by setting up the category structure by which we 
divide up the world into different sorts of objects. Whorf (1956) claimed that Eskimos 
have many words for snow, and thus experience snow in a different way from speakers 
of other languages. His analysis has since been discredited (Pullum, 1991); Inuit, 
Yupik, and other related languages seem to have about the same number of words 
for snow-related concepts as English (consider blizzard, sprinkling, flurries, powder, 
slush, snowbank, snowdrift, etc.). Of course, different languages do carve up the 
world differently. Spanish has two words for "fish," one for the live animal and one 
for the food. English does not make this distinction, but it does have the cow/beef 
distinction. There is no evidence that this means that English and Spanish speakers 
think about the world in fundamentally different ways. 

For our purposes, it is important to remember that the language used to represent 
an agent's internal knowledge is quite different from the external language used to 
communicate with other agents. (See Chapter 22 for the study of communication.) 
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it corresponds to. A sentence does not mean something by itself. This is a difficult concept to 
accept, because we are used to languages like English where the interpretation of most things 
was fixed a long time ago. 

The idea of interpretation is easier to see in made-up languages. Imagine that one spy 
wants to pass a message to another, but worries that the message may be intercepted. The two 
spies could agree in advance on a nonstandard interpretation in which, say, the interpretation of 
"Pope" is a particular piece of microfilm and the interpretation of "Denver" is the pumpkin left 
on the porch, and so forth. Then, when the first spy sends a newspaper clipping with the headline 
"The Pope is in Denver," the second spy will know that the microfilm is in the pumpkin. 

It is possible, in principle, to define a language in which every sentence has a completely 
arbitrary interpretation. But in practice, all representation languages impose a systematic relation- 
ship between sentences and facts. The languages we will deal with are all compositional — the 
meaning of a sentence is a function of the meaning of its parts. Just as the meaning of the math- 
ematical expression x 2 + y 2 is related to the meanings of x 2 and y 2 , we would like the meaning 
of the sentence “£1,4 and Si, 2” to be related to the meanings of “S1.4” and “Si, 2.” It would be 
very strange if "51,4" meant there is a stench in square [1,4] and “Si, 2” meant there is a stench in 
square [1,2], but “Si, 4 and Si, 2” meant that France and Poland drew 1-1 in last week's ice-hockey 
qualifying match. In Section 6.4, we describe the semantics of a simple language, the language 
of propositional logic, that obeys constraints like these. Such constraints make it easy to specify 
a proof theory that respects the semantics. 

Once a sentence is given an interpretation by the semantics, the sentence says that the 
world is this way and not that way. Hence, it can be true or false. A sentence is true under a 
particular interpretation if the state of affairs it represents is the case. Note that truth depends 
both on the interpretation of the sentence and on the actual state of the world. For example, the 
sentence “Sj^” would be true under the interpretation in which it means that there is a stench in 
[1,2], in the world described in Figure 6.2. But it would be false in worlds that do not have a 
stench in [1,2], and it would be false in Figure 6.2 under the interpretation in which it means that 
there is a breeze in [1,2]. 



Inference 

The terms "reasoning" and "inference" are generally used to cover any process by which con- 
clusions are reached. In this chapter, we are mainly concerned with sound reasoning, which 
we will call logical inference or deduction. Logical inference is a process that implements the 
entailment relation between sentences. There are a number of ways to approach the design of 
logical inference systems. We will begin with the idea of a necessarily true sentence. 



Validity and satisfiability 

A sentence is valid or necessarily true if and only if it is true under all possible interpretations in 
all possible worlds, that is, regardless of what it is supposed to mean and regardless of the state 
of affairs in the universe being described. For example, the sentence 
"There is a stench at [1,1] or there is not a stench at [1,1].” 
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is valid, because it is true whether or not "there is a stench in [ 1 , 1 J” is true, and it is true regardless 
ofthe interpretation of "there is a stench in [ 1,1].” In contrast, 

"There is an open area in the square in front of me or there is a wall in the square in 
front of me." 

is not valid by itself, it is only valid under the assumption that every square has either a wall or 
an open area in it. So the sentence 

"lfevery square has either a wall or an open area in it, then there is an open area in 
the square in front of me, or there is a wall in the square in front of me." 
is valid. 6 There are several synonyms for valid sentences. Some authors use the terms analytic^ 
sentences or tautologies for valid sentences. 

satisfiable A sentence is satisfiable if and only if there is some interpretation in some world for 

which it is true. The sentence "there is a wumpus at [1,2]” is satisfiable because there might 
well be a wumpus in that square, even though there does not happen to be one in Figure 6.2. A 
unsatisfiable sentence that is not satisfiable is unsatisfiable. Self-contradictory sentences are unsatisfiable, if 
the contradictoriness does not depend on the mean'iTTgs of the symbols. For example, the sentence 
"There is a wall in front of me and there is no wall in front of me" 
is unsatisfiable. 



Inference in computers 

It might seem that valid and unsatisfiable sentences are useless, because they can only express 
things that are obviously true or false. In fact, we will see that validity and unsatisfiability are 
crucial to the ability of a computer to reason. 

The computer suffers from two handicaps: it does not necessarily know the interpretation 
you are using for the sentences in the knowledge base, and it knows nothing at all about the world 
except what appears in the knowledge base. Suppose we ask the computer if it is OK to move to 
square [2,2]. The computer does not know what OK means, nor does it know what a wumpus or 
a pit is. So it cannot reason informally as we did on page 155. All it can do is see if its knowledge 
base entails the sentence "[2,2] is OK." In other words, the inference procedure has to show that 
the sentence "If KB is true then [2,2] is OK" is a valid sentence. If it is valid, then it does not 
matter that the computer does not know the interpretation you are using or that it does not know 
much about the world — the conclusion is guaranteed to be correct under all interpretations in all 
worlds in which the original KB is true. In Section 6.4, we will give an example of a formal 
procedure for deciding if a sentence is valid. 

What makes formal inference powerful is that there is no limit to the complexity of the 
sentences it can handle. When we think of valid sentences, we usually think of simple examples 
like "The wumpus is dead or the wumpus is not dead." But the formal inference mechanism can 
just as well deal with valid sentences of the form "If KB then FT where KB is a conjunction of 
thousands of sentences describing the laws of gravity and the current state of the solar system, 
and P is a long description of the eventual departure of Pluto from the system. 

6 In these examples, we are assuming that words like "if." "then," "every," "or" and "not" are part ofthe standard syntax 
of the language, and thus are not open to varying interpretation. 
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To reiterate, the great thing about formal inference is that it can be used to derive valid 
conclusions even when the computer does not know the interpretation you are using. The 
computer only reports valid conclusions, which must be true regardless of your interpretation. 
Because you know the interpretation, the conclusions will be meaningful to you, and they are 
guaranteed to follow from your premises. The word “ you ” in this paragraph can be applied 
I - equally to human and computer agents. 



Logics 

To summarize, we can say that a logic consists of the following: 

1 . A formal system for describing states of affairs, consisting of 

(a) the syntax of the language, which describes how to make sentences, and 

(b) the semantics of the language, which states the systematic constraints on how sen- 
tences relate to states of affairs. 

2. The proof theory — a set of rules for deducing the entailments of a set of sentences. 



PROPOSITIONAL 



TEMPORAL LOGIC 



SSEggfjj 1 * 



We will concentrate on two kinds of logic: propositional or Boolean logic, and first-order logic 
(more precisely, first-order predicate calculus with equality). 

In propositional logic, symbols represent whole propositions (facts); for example, D 
might have the interpretation "the wumpus is dead.” which may or may not be a true proposition. 
Proposition symbols can be combined using Boolean connectives to generate sentences with more 
complex meanings. Such a logic makes very little commitment to how things are represented, so 
it is not surprising that it does not give us much mileage as a representation language. 

First-order logic commits to the representation of worlds in terms of objects and predicates 
on objects (i.e., properties of objects or relations between objects), as well as using connectives 
and quantifiers, which allow sentences to be written about everything in the universe at once. 
First-order logic seems to be able to capture a good deal of what we know about the world, and 
has been studied for about a hundred years. We will spend therefore a good deal of time looking 
at how to do representation and deduction using it. 

It is illuminating to consider logics in the light of their ontological and epistemological 
commitments. Ontological commitments have to do with the nature of reality. For example, 
propositional logic assumes that there are facts that either hold or do not in the world. Each fact 
can be in one of two states: true or false. First-order logic assumes more: namely, that the world 
consists of objects with certain relations between them that do or do not hold. Special-purpose 
logics make still further ontological commitments; for example, temporal logic assumes that 
the world is ordered by a set of time points or intervals, and includes built-in mechanisms for 
reasoning about time. 

Epistemological commitments have to do with the possible states of knowledge an agent 
can have using various types of logic. In both propositional and first-order logic, a sentence 
represents a fact and the agent either believes the sentence to be true, believes it to be false, 
or is unable to conclude either way. These logics therefore have three possible states of belief 
regarding any sentence. Systems using probability theory, on the other hand, can have any 
degree of belief, ranging from 0 (total disbelief) to 1 (total belief). For example, a probabilistic 
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wumpus- world agent might believe that the wumpus is in [1,3] with probability 0.75. Systems 
FUZZY LOGIC based on fuzzy logic can have degrees of belief in a sentence, and also allow degrees of truth: 

a fact need not be true or false in the world, but can be true to a certain degree. For example, 
"Vienna is a large city" might be true only to degree 0.6. The ontological and epistemological 
commitments of various logics are summarized in Figure 6.7. '/ 



Language 


Ontological Commitment 
(What exists in the world) 


Epistemological Commitment 
(What an agent believes about facts) 


Propositional logic 
First-order logic 
Temporal logic 
Probability theory 
Fuzzy logic 


facts 

facts, objects, relations 
facts, objects, relations, times 
facts 

degree of truth 


true/false/unknown 
true/false/unknown 
true/false/unknown 
degree of belief 0 ... 1 
degree of belief 0...1 


Figure 6.7 Formal languages and their ontological and epistemological commitments. 



6.4 PROPOSITIONAL LOGIC: A VERY SIMPLE LOGIC 



Despite its limited expressiveness, propositional logic serves to illustrate many of the concepts 
of logic just as well as first-order logic. We will describe its syntax, semantics, and associated 
inference procedures. 

Syntax 

The syntax of propositional logic is simple. The symbols of propositional logic are the logical 
constants True and False, proposition symbols such as P and Q, the logical connectives A, V, •£>, 
=>■, and -i, and parentheses, ( ). All sentences are made by putting these symbols together using 
the following rules: 

• The logical constants True and False are sentences by themselves. 

• A propositional symbol such as P or Q is a sentence by itself. 

• Wrapping parentheses around a sentence yields a sentence, for example, (P A Q). 

• A sentence can be formed by combining simpler sentences with one of the five logical 
connectives: 

A (and). A sentence whose main connective is A, such asFAfQVR), is called a 
conjunction (logic); its parts are the conjuncts. (The A looks like an "A" for “And.”) 
V (or). A sentence using V, such as A V (PA Q), is a disjunction of the disjunctsA 
and (P A Q). (Historically, the V comes from the Latin “vel,” which means "or." For 
most people, it is easier to remember as an upside-down and.) 



CONJUNCTION 

(LOGIC) 

DISJUNCTION 
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IMPLICATION 

PREMISE 

CONCLUSION 

EQUIVALENCE 

-negation 



atomic SENTENCES 

Se'|JE| E El 

LITERAL 



=> (implies). A sentence such as (FAQ) =>• R is called an implication (or conditional). 

Its premise or antecedent is P A Q, and its conclusion or consequent is R. Impli- 
cations are also known as rules or if-then statements. The implication symbol is 
sometimes written in other books as D or — . 

-t=> (equivalent). The sentence (P A Q) <=> (Q A P) is an equivalence (also called a 
biconditional). 

-> (not). A sentence such as -<P is called the negation of P. All the other connectives 
combine two sentences into one; -i is the only connective that operates on a single 
sentence. 

Figure 6.8 gives a formal grammar of propositional logic; see page 854 if you are not fa- 
miliar with the BNF notation. The grammar introduces atomic sentences, which in propositional 
logic consist of a single symbol (e.g., P), and complex sentences, which contain connectives or 
parentheses (e.g., P A Q). The term literal is also used, meaning either an atomic sentences or a 
negated atomic sentence. 



Sentence — Atomic-Sentence ComplexSentence 
Atomic-Sentence — True I False 

I P Q R ■■■ 

ComplexSentence — ( Sentence ) 

| Sentence Connective Sentence 
| -i Sentence 

Connective — A I V | <S>| => 



Figure 6.8 A BNF (Backus-Naur Form) grammar of sentences in propositional logic. 



Strictly speaking, the grammar is ambiguous — a sentence such as P A Q V R could be 
parsed as either (P A Q) V R or as P A (Q V R). This is similar to the ambiguity of arithmetic 
expressions such as P + Q x R, and the way to resolve the ambiguity is also similar: we pick an 
order of precedence for the operators, but use parentheses whenever there might be confusion. 
The order of precedence in propositional logic is (from highest to lowest): A, V, =>, and -O. 

Flence, the sentence 

-■PVgAP => S 



is equivalent to the sentence 
((-nP)V(QAfl)) =*• S. 
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Semantics 

The semantics of propositional logic is also quite straightforward. We define it by specifying 
the interpretation of the proposition symbols and constants, and specifying the meanings of the 
logical connectives. 

A proposition symbol cari mean whatever you want. That is, its interpretation can be any 
arbitrary fact. The interpretation of P might be the fact that Paris is the capital of France or that 
the wumpus is dead. A sentence containingjust a proposition symbol is satisfiablebut not valid: 
it is true just when the fact that it refers to is the case. 

With logical constants, you have no choice; the sentence True always has as its interpretation 
the way the world actually is — the true fact. The sentence False always has as its interpretation 
the way the world is not. 

A complex sentence has a meaning derived from the meaning of its parts. Each connective^ 
can be thought of as a function. Just as addition is a function that takes two numbers as input 
and returns a number, so and is a function that takes two truth values as input and returns a truth 
value. We know that one way to define a function is to make a table that gives the output value 
for every possible input value. For most functions (such as addition), this is impractical because 
of the size of the table, but there are only two possible truth values, so a logical function with 
two arguments needs a table with only four entries. Such a table is called a truth table. We give 
truth tables for the logical connectives in Figure 6.9. To use the table to determine, for example, 
the value of True V False, first look on the left for the row where P is true and Q is false (the third 
row). Then look in that row under the P V Q column to see the result: True. 



p 


Q 


->P 


PAQ 


PVQ 


P => Q 


P&Q 


False 


False 


True 


False 


False 


True 


True 


False 


True 


True 


False 
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False ' 


False 


False 


True 


False' 


False 


True 


True 


False 


True 


True 


True 


True 


Figure 6.9 Truth tables for the five logical connectives. J 



Truth tables define the semantics of sentences such as True A True. Complex sentences 
such as (P V 0 A ->S are defined by a process of decomposition: first, determine the meaning of 
(PA ® and of -i S, and then combine them using the definition of the A function. This is exactly 
analogous to the way a complex arithmetic expression such as (px q) -t — ,v is evaluated. 

The truth tables for "and," "ar,” and "not" are in close accord with our intuitions about the 
English words. The main point of possible confusion is that P V Q is true when either or both P 
and Q are true. There is a different connective called "exclusive or" (“xor” for short) that gives 
false when both disjuncts are true. 7 There is no consensus on the symbol for exclusive or; two 
choices are V and $. 

In some ways, the implication connective => is the most important, and its truth table might 
seem puzzling at first, because it does not quite fit our intuitive understanding of "P implies Q” 



7 Latin has a separate word, aut, for exclusive or. 
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or "if P then Q. " For one thing, propositional logic does not require any relation of causation or 
relevance between P and Q. The sentence "5 is odd implies Tokyo is the capital of Japan" is a true 
sentence of propositional logic (under the normal interpretation), even though it is a decidedly 
odd sentence of English. Another point of confusion is that any implication is true whenever its 
antecedent is false. For example, "5 is even implies Sam is smart" is true, regardless of whether 
Sam is smart. This seems bizarre, but it makes sense if you think of "P => Q" as saying, "If P 
is true, then I am claiming that Q is true. Otherwise I am making no claim." 

Validity and inference 

Truth tables can be used not only to define the connectives, but also to test for valid sentences. 
Given a sentence, we make a truth table with one row for each of the possible combinations of 
truth values for the proposition symbols in the sentence. For each row, we can calculate the truth 
value of the entire sentence. If the sentence is true in every row, then the sentence is valid. For 
example, the sentence 

((PVH)A -ntf) => P 

is valid, as can be seen in Figure 6.10. We include some intermediate columns to make it clear 
how the final column is derived, but it is not important that the intermediate columns are there, as 
long as the entries in the final column follow the definitions of the connectives. Suppose P means 
that there is a wumpus in [1,3] and //means there is a wumpus in [2,2]. If at some point we learn 
(P V H) and then we also leam — ■//, then we can use the valid sentence above to conclude that P 
is true — that the wumpus is in f 1,3). 



p 


H 


PVH 


(PVH)A ->// 
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False 


True 


Figure 6.10 


Truth table showing validity of a complex sentence. 





This is important. It says that if a machine has some premises and a possible conclusion, 
it can determine if the conclusion is true. It can do this by building a truth table for the sentence 
Premises => Conclusion and checking all the rows. If every row is true, then the conclusion is 
entailed by the premises, which means that the fact represented by the conclusion follows from 
the state of affairs represented by the premises. Even though the machine has no idea what the 
conclusion means, the user could read the conclusions and use his or her interpretation of the 
proposition symbols to see what the conclusions mean — in this case, that the wumpus is in [ 1,3]. 
Thus, we have fulfilled the promise made in Section 6.3. 

It will often be the case that the sentences input into the knowledge base by the user refer 
to a world to which the computer has no independent access, as in Figure 6.1 1, where it is the 
user who observes the world and types sentences into the computer. It is therefore essential 
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that a reasoning system be able to draw conclusions that follow from the premises, regardless 
of the world to which the sentences are intended to refer. But it is a good idea for a reasoning 
system to follow this principle in any case. Suppose we replace the "user" in Figure 6.1 1 with 
a camera-based visual processing system that sends input sentences to the reasoning system. It 
makes no difference! Even though the computer now has "direct access" to the world, inference 
can still take place through direct operations on the syntax of sentences, without any additional 
information as to their intended meaning. 




Models 

model Any world in which a sentence is true under a particular interpretation is called a model of that 

sentence under that interpretation. Thus, the world shown in Figure 6.2 is a model of the sentence 
“£12” under the interpretation in which it means that there is a stench in [1,2]. There are many 
other models of this sentence — you can make up a world that does or does not have pits and gold 
in various locations, and as long as the world has a stench in [1,2], it is a model of the sentence. 
The reason there are so many models is because “51,2'’ makes a very weak claim about the world._ 
The more we claim (i.e., the more conjunctions we add into the knowledge base), the fewer the 
models there will be. 

Models are very important in logic, because, to restate the definition of entailment, a 
I A sentence a is entailed by a knowledge base KB if the models of KB are all models of a. If this is 

the case, then whenever KB is true, a must also be true. 

In fact, we could define the meaning of a sentence by means of set operations on sets of 
models. For example, the set of models of P A Q is the intersection of the models of P and the 
models of Q. Figure 6.12 diagrams the set relationships for the four binary connectives. 

We have said that models are worlds. One might feel that real worlds are rather messy 
things on which to base a formal system. Some authors prefer to think of models as mathematical 
objects. In this view, a model in propositional logic is simply a mapping from proposition symbols 
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directly to truth and falsehood, that is, the label for a row in a truth table. Then the models of 
a sentence are just those mappings that make the sentence true. The two views can easily be 
reconciled because each possible assignment of true and false to a set of proposition symbols 
can be viewed as an equivalence class of worlds that, under a given interpretation, have those 
truth values for those symbols. There may of course be many different "real worlds" that have 
the same truth values for those symbols. The only requirement to complete the reconciliation is 
that each proposition symbol be either true orfalse in each world. This is, of course, the basic 
ontological assumption of propositional logic, and is what allows us to expect that manipulations 
of symbols lead to conclusions with reliable counterparts in the actual world. 

Rules of inference for propositional logic 

The process by which the soundness of an inference is established through truth tables can be 
extended to entire classes of inferences. There are certain patterns of inferences that occur over 
and over again, and their soundness can be shown once and for all. Then the pattern can be 
NFERENCE RULE captured in what is called an inference rule. Once a rule is established, it can be used to make 
inferences without going through the tedious process of building truth tables. 

We have already seen the notation a h 8 to say that 6 can be derived from a by inference. 
There is an alternative notation, 



P 

which emphasizes that this is not a sentence, but rather an inference rule. Whenever something in 
the knowledge base matches the pattern above the line, the inference rule concludes the premise 
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below the line. The letters a, i3, etc., are intended to match any sentence, not just individual 
proposition symbols. If there are several sentences, in either the premise or the conclusion, they 
are sep arated by commas. Figure 6.13 gives a list of seven commonly used inference rules. 

An inference rule is sound if the conclusion is true in all cases where the premises are true. 
To verify soundness, we therefore construct a truth table with one line for each possible model 
of the proposition symbols in the premise, and show that in all models where the premise is true, 
the conclusion is also true. Figure 6.14 shows the truth table for the resolution rule. 



<C> Modus Ponens or Implication-Elimination: (From an implication and the 
premise of the implication, you can infer the conclusion.) 
a => (3, a 

P 

0 And-Elimination: (From a conjunction, you can infer any of the conjuncts.) 
c*l A 02 A ...Ao„ 



<C> And-Introduction: (From a list of sentences, you can infer their conjunction.) 

01,02, , Q„ 

a i A »2 A ... A 

0 Or-Introduction: (From a sentence, you can infer its disjunction with anything 
else at all.) 



o, Vo 2 V...Vo„ 

0 Double-Negation Elimination: (From a doubly negated sentence, you can infer 
a positive sentence.) 



0 Unit Resolution: (From a disjunction, if one of the disjuncts is false, then you 
can infer the other one is true.) 
a V /?, ->li 



a 

O’ Resolution: (This is the most difficult. Because 0 cannot be both true and false, 
one of the other disjuncts must be true in one of the premises. Or equivalently, 
implication is transitive.) 



aVp, 7 

a Vi 



or equivalently 



-,a =>■/?, /3 => j 

-•a => 7 



Figure 6.13 Seven inference rules for propositional logic. The unit resolution rule is a special 
case of the resolutionxule^ which in turn is a special case of the full resolution rule for first-order 
logic discussed in Chapter 9. 
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True 


True 


True 


Figure 6.14 A truth table demonstrating the soundness of the resolution inference rule. We 
have underlined the rows where both premises are true. 



As we mentioned above, a logical proof consists of a sequence of applications of inference 
rules, starting with sentences initially in the KB, and culminating in the generation of the sentence 
whose proof is desired. To prove that P follows from (P V H) and — ■//, for example, we simply 
require one application of the resolution rule, with a as P, i3 as H, and 7 empty. The job of an 
inference procedure, then, is to construct proofs by finding appropriate sequences of applications 
of inference rules. 



Complexity of propositional inference 

The truth-table method of inference described on page 169 is complete, because it is always 
possible to enumerate the 2" rows of the table for any proof involving n proposition symbols. 
On the other hand, the computation time is exponential in n, and therefore impractical. One 
might wonder whether there is a polynomial-time proof procedure for propositional logic based 
on using the inference rules from Section 6.4. 

In fact, a version of this very problem was the first addressed by Cook (1971) in his theory 
of NP-completeness. (See also the appendix on complexity.) Cook showed that checking a set of 
sentences for satisfiability is NP-complete, and therefore unlikely to yield to a polynomial-time 
algorithm. However, this does not mean that all instances of propositional inference are going to 
take time proportional to 2". In many cases, the proof of a given sentence refers only to a small 
subset of the KB and can be found fairly quickly. In fact, as Exercise 6.15 shows, really hard 
problems are quite rare. 

The use of inference rules to draw conclusions from a knowledge base relies implicitly 
on a general property of certain logics (including propositional and first-order logic) called 
MONOTONICITY monotonicity. Suppose that a knowledge base KB entails some set of sentences. A logic is 
monptonic if when we add some new sentences to the knowledge base, all the sentences entailed 
by the original KB are still entailed by the new larger knowledge base. Formally, we can state 
the property of monotonicity of a logic as follows: 
if KB X (= a then (KB t U KB 2 ) |= a 

This is true regardless of the contents of KB 2 — it can be irrelevant or even contradictory to KB\ . 
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It is fairly easy to show that propositional and first-order logic are monotonic in this sense; 
one can also show that probability theory is not monotonic (see Chapter 14). An inference rule" 
LOCAL such as Modus Ponens is local because its premise need only be compared with a small portion 

of the KB (two sentences, in fact). Were it not for mono tonicity, we could not have any local 
inference rules because the rest of the KB might affect the soundness of the inference. This 
would potentially cripple any inference procedure. 

There is also a useful class of sentences for which a polynomial-time inference procedure 
HORN SENTENCES exists. This is the class called Horn sentences . 8 A Horn sentence has the form: 

P, A P 2 A . . . A P n => Q 

where the Pj and Q are nonnegated atoms. There are two important special cases: -First, when 
Q is the constant False, we get a sentence that is equivalent to -<P] V ... V ->P„. Second, when 
n = 1 and P\ = True, we get True => Q, which is equivalent to the atomic sentence Q. Not every 
knowledge base can be written as a collection of Horn sentences, but for those that can, we can use 
a simple inference procedure: apply Modus Ponens wherever possible until no new inferences 
remain to be made. We discuss Horn sentences and their associated inference procedures in more 
detail in the context of first-order logic (Section 9.4). 



6.5 AN AGENT FOR THE WUMPUS WORLD 



In this section, we show a snapshot of a propositional logic agent reasoning about the wumpus 
world. We assume that the agent has reached the point shown in Figure 6.4(a), repeated here as 
Figure 6.15, and show how the agent can conclude that the wumpus is in [1,3]. 



The knowledge base 

On each turn, the agent's percepts are converted into sentences and entered into the knowledge 
base, along with some valid sentences that are entailed by the percept sentences. Let us assume 
that the symbol 9 5), 2, for example, means "There is a stench in [1,2].” Similarly, B 2 ,\ means 
"There is a breeze in [2,1].” At this point, then, the knowledge base contains, among others, the 
percept sentences 

-iSu ifli.i 

-1S2.1 B 2 , 1 

Si, 2 ~'Bia 

In addition, the agent must start out with some knowledge of the environment. For example, the 
agent knows that if a square has no smell, then neither the square nor any of its adjacent squares 



8 Also known as Horn clauses. The name honors the logician Alfred Horn. 

9 The subscripts make these symbols look like they have some kind of internal structure, but do not let that mislead you. 
We could have used Q or StenchOneTwoinstead of S1.2, but we wanted symbols that are both mnemonic and succinct. 
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0 = Agent 
B = Breeze 
G = Glitter, Gold 
OK = Safe square 
P = Pit 
S = Stench 
V = Visited 
W = Wumpus 



Figure 6.15 The agent's knowledge after the third move. The current percept is 
[Stench, None, None, None, None], 



can house a wumpus. The agent needs to know this for each square in the world, but here wejust 
show sentences for three relevant squares, labeling each sentence with a rule number: 

Ri : -iSu => ->W U A-Wi .2 A-1W2.1 
R 2 : -d> 2 , 1 => -.W1.1 A - 1 W 2 , 1 A — .W 2i2 A ->W 3 , 1 
Ry- -5i , 2 => — 'W - !,! A nf, 2 A ->W 2>2 A ->Wi,3 
Another useful fact is that if there is a stench in [1 ,2], then there must be a wumpus in [1,2] or in 
one or more of the neighboring squares. This fact can be represented by the sentence 
R 4 : 5 1 -2 =k Wi, 3 VW,, 2 V W 2 , 2 V W,., 



Finding the wumpus 

Given these sentences, we will now show how an agent can mechanically conclude W\t,. All the 
agent has to do is construct the truth table for KB =>■ W\ t, to show that this sentence is valid. 
There are 12 propositional symbols, 10 so the truth table will have 2 12 = 4096 rows, and every 
row in which the sentence KB is true also has W 1,3 true. Rather than show all 4096 rows, we use 
inference rules instead, but it is important to recognize that we could have done it in one (long) 
step just by following the truth-table algorithm. 

First, we will show that the wumpus is not in one of the other squares, and then conclude 
by elimination that it must be in [ 1 ,3] : 

1. Applying Modus Ponens with -iSu and the sentence labelled R\ , we obtain 

->W U A -nWu A -.W 2 ,, 

2. Applying And-Elimination to this, we obtain the three separate sentences 

-W.,i -W.,2 -WA, 

10 The 12 symbols are 5 u ,5 2 , i , Si, 2, (%, W l3 , W 2 ,i,W v ,W3,t,W^.B h i,B 2 ,i,B,a. 
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3. Applying Modus Ponens to ->S 2 .i and the sentence labelled R2, and then applying And- 
Elimination to the result, we obtain the three sentences 

-1W2.2 -'W'2,1 ->W 3 , 1 

4. Applying Modus Ponens to Si ,2 and the sentence labelled R4, we obtain 

W ,, 3 V Wi ,2 V VP 2 , 2 v'W U 

5. Now we apply the unit resolution rule, where a is W1.3 V W\^W W 3 ,i and 0 is VP 11 . (We 
derived — > in step 2.) Unit resolution yields 

W13 V VP] 2 v W 2j2 

6. Applying unit resolution again with W13V PP 1 ,2 as a and VP 2 , 2 as 0 (~iW 2 , 2 was derived in 
step 3), we obtain 

IP, ,3 v 1P|,2 

7. Finally, one more resolution with VP| 2 as Q and VP] j2 as 0 (we derived -iW ) j2 in step 2) 
gives us the answer we want, namely, that the wumpus is in [1,3]: 

W 13 



Translating knowledge into action 

We have shown how propositional logic can be used to infer knowledge such as the whereabouts 
of the wumpus. But the knowledge is only useful if it helps the agent take action. To do that, 
we will need additional rules that relate the current state of the world to the actions the agent 
should take. For example, if the wumpus is in the square straight ahead, then it is a bad idea to 
execute the action Forward. We can represent this with a series of rules, one for each location 
and orientation in which the agent might be. Here is the rule for the case where the agent is in 
[1,1] facing east: 

A u AEast^A VP 2 j => -> Forward 

Once we have these rules, we need a way to ASK the knowledge base what action to take. 
Unfortunately, propositional logic is not powerful enough to represent or answer the question 
"what action should I take?” but it is able to answer a series of questions such as "should I go 
forward?" or "should I turn right?" That means that the algorithm for a knowledge-based agent 
using propositional logic would be as in Figure 6.16. 



Problems with the propositional agent 

Propositional logic allows us to get across all the important points about what a logic is and how 
it can be used to perform inference that eventually results in action. But propositional logic is so 
weak that it really cannot handle even a domain as simple as the wumpus world. 

The main problem is that there are just too many propositions to handle. The simple rule 
"don’t go forward if the wumpus is in front of you" can only be stated in propositional logic by 
a set of 64 rules (16 squares x 4 orientations for the agent). Naturally, it just gets worse if the 
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function Propositional-KB-Agent( percept) returns an action 
static: KB, a knowledge base 

t, a counter, initially 0, indicating time 

Tell(0, Make-Percept-Sentence( percept, t)) 
for each action in the list of possible actions do 
if Ask( KB. M a ke- Action -Query! r, action)) then 
t—t+ 1 
return action 



Figure 6.16 A knowledge-based agent using propositional logic. 



world is larger than a 4 x 4 grid. Given this multiplication of rules, it will take thousands of rules 
to define a competent agent. The problem is not just that it is taxing to write the rules down, but 
also that having so many of them slows down the inference procedure. Remember that the size 
of a truth table is 2", where n is the number of propositional symbols in the knowledge base. 

Another problem is dealing with change. We showed a snapshot of the agent reasoning at 
a particular point in time, and all the propositions in the knowledge base were true at that time. 
But in general the world changes over time. When the agent makes its first move, the proposition 
Ai,i becomes false and A2 ,i becomes true. But it may be important for the agent to remember 
where it was in the past, so it cannot just forget Au . To avoid confusion, we will need different 
propositional symbols for the agent's location at each time step. This causes difficulties in two 
ways. First, we do not know how long the game will go on, so we do not know how many of these 
time-dependent propositions we will need. Second, we will now have to go back and rewrite 
time-dependent versions of each rule. For example, we will need 
Ai’j A East 0 A => -> Forward 0 
A\ , A East\ A W\ i =>- -■ Forward 1 
zlf, A East a A Wf'i => -> Forward 2 

A? | A North^ A W° 2 => --'Forward 0 
A] , ANorth\A W \ , => -> Forward 1 
A, , A North\ A Wf 2 => -> Forward 2 



where the superscripts indicate times. If we want the agent to run for 100 time steps, we will 
need 6400 of these rules, just to say one should not go forward when the wumpus is there. 

In summary, the problem with propositional logic is that it only has one representational 
device: the proposition. In the next chapter, we will introduce first-order logic, which can 
represent objects and relations between objects in addition to propositions. In first-order logic, 
the 6400 propositional rules can be reduced to one. 
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6.6 SUMMARY 



We have introduced the idea of a knowledge-based agent, and showed how we can define a logic 
with which the agent can reason, about the world and be guaranteed to draw correct conclusions, 
given correct premises. We have also showed how an agent can turn this knowledge into action. 
The main points are as follows: 

• Intelligent agents need knowledge about the world in order to reach good decisions. 

• Knowledge is contained in agents in the form of sentences in a knowledge representation 
language that are stored in a knowledge base. 

• A knowledge-based agent is composed of a knowledge base and an inference mechanism. 
It operates by storing sentences about the world in its knowledge base, using the inference 
mechanism to infer new sentences, and using them to decide what action to take. 

• A representation language is defined by its syntax and semantics, which specify the 
structure of sentences and how they relate to facts in the world. 

• The interpretation of a sentence is the fact to which it refers. If it refers to a fact that is 
part of the actual world, then it is true. 

• Inference is the process of deriving new sentences from old ones. We try to design sound 
inference processes that derive true conclusions given true premises. An inference process 
is complete if it can derive all true conclusions from a set of premises. 

• A sentence that is true in all worlds under all interpretations is called valid. If an implication 
sentence can be shown to be valid, then we can derive its consequent if we know its premise. 
The ability to show validity independent of meaning is essential. 

• Different logics make different commitments about what the world is made of and what 
kinds of beliefs we can have regarding facts. 

• Logics are useful for the commitments they do not make, because the lack of commitment 
gives the knowledge base writer more freedom. 

• Propositional logic commits only to the existence of facts that may or may not be the case in 
the world being represented. It has a simple syntax and semantics, but suffices to illustrate 
the process of inference. 

• Propositional logic can accommodate certain inferences needed by a logical agent, but 
quickly becomes impractical for even very small worlds. 



BIBLIOGRAPHICAL AND HISTORICAL NOTES 

Logic had its origins in ancient Greek philosophy and mathematics. Logical principles — 
principles connecting the syntactic structure of sentences with their truth and falsity, their 
meaning, or the validity of arguments in which they figure — can be found in scattered loca- 
tions in the works of Plato (428-348 B.C.). The first known systematic study of logic was carried 
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out by Aristotle, whose work was assembled by his students after his death in 322 B.C. as a 
treatise called the Organon, the first systematic treatise on logic. However, Aristotle's logic 
was very weak by modern standards; except in a few isolated instances, he did not take account 
of logical principles that depend essentially on embedding one entire syntactic structure within 
another structure of the same type, in the way that sentences are embedded within other sentences 
in modern propositional logic. Because of this limitation, there was a fixed limit on the amount 
of internal complexity within a sentence that could be analyzed using Aristotelian logic. 

The closely related Megarian and Stoic schools (originating in the fifth century B.C. and 
continuing for several centuries thereafter) introduced the systematic study of implication and 
other basic constructs still used in modem propositional logic. The Stoics claimed that their logic 
was complete in the sense of capturing all valid inferences, but what remains is too fragmentary 
to tell. A good account of the history of Megarian and Stoic logic, as far as it is known, is given 
by Benson Mates (1953). 

The ideas of creating an artificial formal language patterned on mathematical notation 
in order to clarify logical relationships, and of reducing logical inference to a purely formal 
and mechanical process, were due to Leibniz (1646-1716). Leibniz's own mathematical logic, 
however, was severely defective, and he is better remembered simply for introducing these ideas 
as goals to be attained than for his attempts at realizing them. 

George Boole (1847) introduced the first reasonably comprehensive and approximately 
correct system of logic based on an artificial formal language with his book The Mathematical 
Analysis of Logic. Boole's logic was closely modeled on the ordinary algebra of real numbers. 
Boole’s system subsumed the main parts of Aristotelian logic and also contained a close analogue 
to modem propositional logic. Although Boole's system still fell short of full propositional logic, 
it was close enough that other 19th-century writers following Boole could quickly fill in the gaps. 
The first comprehensive exposition of modern propositional logic (and first-order logic) is found 
in Gottlob Frege's (1879) Begriffschrift ' Concept Writing" or "Conceptual Notation"). 

Truth tables as a method of testing the validity or unsatisfiability of sentences in the lan- 
guage of propositional logic were independently introduced simultaneously by Ludwig Wittgen- 
stein ( 1 922) and by Emil Post (1921). (Asa method of explaining the meanings of propositional 
connectives, tmth tables go back to Philo ofMegara.) 

Quine (1982) describes “truth-value analysis," a proof method closely resembling truth 
tables but more efficient because, in effect, it can handle multiple lines of the truth table si- 
multaneously. Wang (1960) takes a general proof method for first-order logic designed by 
Gentzen (1934) and selects a convenient and efficient subset of inference mles for use in a 
tree-based procedure for deciding validity in propositional logic. 

John McCarthy's (1968) paper "Programs with Common Sense" promulgated the notion 
of agents that use logical reasoning to mediate between percepts and actions. This paper was the 
first to make this conception widely known, although it draws on much earlier work (McCarthy, 
1 958). The 1968 paper is also called the "Advice Taker" paper because it introduces ahypothetical 
program by that name which uses logic to enable its designers to communicate useful knowledge to 
it without having to write further directly executable computer code. Alien Newell's ( 1 982) article 
"The Knowledge Level” focuses on the use of logic by agent designers to describe the knowledge 
that is, in effect, being used by the agents they are designing, whether or not the agents themselves 
use explicit logical formulas to represent this knowledge internally. This theme of Newell's was 
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hinted at in 1 943 by the psychologist Kenneth Craik, who writes, "My hypothesis then is that 
thought models, or parallels, reality — that its essential feature is not ‘the mind,’ ‘the self,’ ‘sense- 
data,’ nor propositions but symbolism, and that this symbolism is largely of the same kind as that 
which is familiar to us in mechanical devices which aid thought and calculation . . (Craik, 1943). 
Further work along these lines has been done by Rosenschein and Kaelbling (Rosenschein, 1 985; 
Kaelbling and Rosenschein, 1990). Rosenschein and Genesereth (1987) have researched the 
problem of cooperative action among agents using propositional logic internally to represent 
the world. Gabbay (1991) has explored extensions to standard logic to enhance guidance of 
reasoning and retrieval from large knowledge bases. 



EXERCISES 



6.1 We said that truth tables can be used to establish the validity of a complex sentence. Show 
how they can be used to decide if a given sentence is valid, satisfiable, or unsatisfiable. 

6.2 Use truth tables to show that the following sentences are valid, and thus that the equivalences 
hold. Some of these equivalence rules have standard names, which are given in the right column. 



PA (Q AR) 




(P A 2) A ft 


Associativity of conjunction 


PV (2 V R) 


<s> 


(PVOVfi 


Associativity of disjunction 


P/\Q 


o- 


GAP 


Commutativity of conjunction 


P\JQ 


o 


2vP 


Commutativity of disjunction 


P A (QVR) 




(P A 0 V (PA ft) 


Distributivity of A over V 


PV(2Aft) 




(PVOA(PVft) 
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-(PAG) 




-iPV-ig 


de Morgan's Law 


-ffv 0 




-<P A —iQ 


de Morgan’s Law 


P => Q 




-e => -p 


Contraposition 


■nP 




p 


Double Negation 


0 => Q 


<£> 


-pve 




. _P 2 




(P =► 0 A (Q =► P) 




P Q 




(P A 0 V (-iPA -i0 




PA ->P 




False 




py^p 


o 


True 





6.3 Look at the following sentences and decide for each if it is valid, unsatisfiable, or neither. 
Verify your decisions using truth tables, or by using the equivalence rules of Exercise 6.2. Were 
there any that initially confused you? 

a. Smoke => Smoke 

b. Smoke => Fire 

c. (Smoke => Fire ) => (-> Smoke => -iFire) 

d. Smoke V Fire V -iFire 
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e. ({Smoke A Heat) => Fire) O ((Smoked Fire) V (Heat =>■ Fire)) 

f. (Smoke Fire) => ((Smoke A Heat) => Fire) 

g. Big V Dumb V (Big => Dumb) 

h. (Big A Dumb) M-^Dumb 

6.4 Is the sentence "Either 2 + 2 = 4 and it is raining, or 2 + 2 = 4 and it is not raining" making 
a claim about arithmetic, weather, or neither? Explain. 

6.5 (Adapted from (Barwise and Etchemendy, 1993).) Given the following, can you prove that 
the unicorn is mythical? How about magical? Horned? 

* ' If the unicorn is mythical, then it is immortal, but if it is not mythical, then it is a 
mortal mammal. If the unicorn is either immortal or a mammal, then it is horned. 

The unicorn is magical if it is horned. 

6.6 What ontological and epistemological commitments are made by the language of real- 
number arithmetic? 

6.7 Consider a world in which there are only four propositions, A, B, C, and D. How many 
models are there for the following sentences? 

a. AAB 

b. A VB 

C. A ABAC 

6.8 We have defined four different binary logical connectives. 

a. Are there any others that might be useful? 

b. How many binary connectives can there possibly be? 

c. Why are some of them not very useful? 

6.9 Some agents make inferences as soon as they are told a new sentence, while others wait until 
they are asked before they do any inferencing. What difference does this make at the knowledge 
level, the logical level, and the implementation level? 

6.10 We said it would take 64 propositional logic sentences to express the simple rule "don't 
go forward if the wumpus is in front of you." What if we represented this fact with the single rule 

WumpusAhead => -i Forward 

Is this feasible? What effects does it have on the rest of the knowledge base? 

6.11 Provide a formal syntax, semantics, and proof theory for algebraic equations including 
variables, numbers, +, — , x, and -+. You should be able to provide inference steps for most 
standard equation manipulation techniques. 
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6.12 (Adapted from (Davis, 1990).) Jones, Smith, and Clark hold the jobs of programmer, 
knowledge engineer, and manager (not necessarily in that order). Jones owes the programmer 
$10. The manager's spouse prohibits borrowing money. Smith is not married. Your task is to 
figure out which person has which job. 

Represent the facts in propositional logic. You should have nine propositional symbols to 
represent the possible person/job assignments. For example, you might use the symbol SM to 
indicate that Smith is the manager. You do not need to represent the relation between owing and 
borrowing, or being married and having a spouse; you can just use these to draw conclusions (e.g, 
from "Smith is not married" and "the manager's spouse" we know that Smith can't be the manager, 
which you can represent as ~^SM). The conjunction of all the relevant facts forms a sentence 
which you can call KB. The possible answers to the problem are sentences like JP ASK A CM. 
There are six such combinations of person/job assignments. Solve the problem by showing that 
only one of them is implied by KB, and by saying what its interpretation is. 



6.13 What is the performance score that one could expect from the optimal wumpus world 
agent? Design an experiment wherein you look at all possible 4x4 wumpus worlds (or a random 
sample of them if there are too many), and for each one determine the shortest safe path to pick 
up the gold and return to start, and thus the best score. This gives you an idea of the expected 
performance score for an ideal omniscient agent. How could you determine the expected score 
for an optimal non-omniscient agent? 



6.14 Implement a function VALIDITY that takes a sentence as input and returns either valid, 
satisfiable, or unsatisfiable. Use it to answer the questions in Exercise 6.12. You will need 
to define an implementation-level representation of sentences. The cleanest way to do this is 
to define an abstract data type for compound sentences. Begin by writing EVAL-TRUTH as a 
recursive function that takes a sentence and an assignment of truth values to proposition symbols, 
and returns true or false. Then call EVAL-TRUTH for all possible assignments of truth values to 
the proposition symbols. 



6.15 SAT is the abbreviation for the satisfiability problem: given a propositional sentence, 
determine if it is satisfiable, and if it is, show which propositions have to be true to make the 
sentence true. 3SAT is the problem of finding a satisfying truth assignment for a sentence in a 
special format called 3-CNF, which is defined as follows: 

• A literal is a proposition symbol or its negation (e.g., P or -i P). 

• A clause is a disjunction of literals; a 3-clause is a disjunction of exactly 3 literals (e.g., 
FV0V-.fi), 

• A sentence in CNF or conjunctive normal form is a conjunction of clauses; a 3-CNF 
sentence is a conjunction of 3-clauses. 

For example, 

(PYQY -.5) A (->P V 2 V R) A (~>PY ->R V ->S) A (P V ->S V T) 
is a 3-CNF sentence with four clauses and five proposition symbols. 

In this exercise, you will implement and test GSAT, an algorithm for solving SAT problems 
that has been used to investigate how hard 3SAT problems are. GSAT is a random-restart, hill- 
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climbing search algorithm. The initial state is a random assignment of true and false to the 
proposition symbols. For example, for the preceding 3-CNF sentence, we might start with P and 
Q false and R, S, and T true. 

The evaluation function measures the number of satisfied clauses, that is, clauses with at 
least one true disjunct. Thus, the initial state gets an evaluation of 3, because the second, third, 
and fourth clauses are true. If there are n proposition symbols, then there are n operators, where 
each operator is to change the truth assignment for one of the symbols. As a hill-climbing search, 
we always use the operator that yields the best evaluation (randomly choosing one if there are 
several equally good operators). Our example 3-CNF sentence is solved in one step, because 
changing S from true to false yields a solution. As with a random-restart algorithm, unless we 
find a solution after a certain amount of hill-climbing, we give up and start over from a new 
random truth assignment. After a certain number of restarts, we give up entirely. The complete 
algorithm is shown in Figure 6.17. 



function Gs /^(sentence, max-resta rts , max-climbs) returns a truth assignment or failure 

for < — 1 to max-restarts do 

A — A randomly generated truth assignment 
for) — 1 to max-climbs do 

if A satisfies sentence then return A 
A — a random choice of one of the best successors of A 
end 

end 

return failure 



Figure 6.17 The Gsat algorithm for satisfiability testing. The successors of an assignment A 
are truth assignment with one symbol flipped. A "best assignment" is one that makes the most 
clauses true. 



Answer the following questions about the algorithm: 

a. Is the GSAT algorithm sound? 

b. Is it complete? 

c. Implement GSAT and use it to solve the problems in Exercise 6.3. 

d. Use GSAT to solve randomly generated 3SAT problems of different sizes. There are two 
key parameters: N, the number of propositional symbols, and C, the number of clauses. 
We will investigate the effects of the ratio C/N on the execution time of GSAT. With N 
fixed at 20, make a graph of the median execution time versus C/N for C/N from 1 to 10. 
(The median is a better statistic than the mean because one or two outliers can really throw 
off the mean.) Use N as the value of max-restarts and 5N as the value of max-climbs. 

e. Repeat for other values of N as time permits. 

f. What can you conclude about the difficulty of 3SAT problems for different values of C, N , 
and the ratio C/N? 
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See Selman et al. ( 1 992) for more on GS AT. They present an implementation of GSAT that solves 
even the hardest 3SAT problems with N = 70 in under a second. GSAT can be used to solve a 
wide variety of problems by constructing a reduction from each class of problems to 3SAT. It 
is so efficient that it often outperforms special-purpose algorithms that are expertly designed for 
specific problems. 

6.16 Consider the problem of designing a logical agent for the wumpus world using a Boolean 
circuit — that is, a collection of logic gates connecting the inputs (percept values) to outputs 
(action values). 

a. Explain why you would need flip-flops. 

b. Give an order-of-magnitude estimate of how many gates and flip-flops would you need. 




~f FIRST-ORDER LOGIC 



In which we introduce a logic that is sufficientfor building knowledge-based agents. 



HRST-ORDER LOGIC 

OBJECTS 

PROPERTIES 

RELATIONS 

FUNCTIONS 



In Chapter 6, we showed how a knowledge-based agent could represent the world in which it 
operates and use those representations to deduce what actions to take. We used propositional 
logic as our representation language because it is one of the simplest languages that demonstrates 
all the important points. Unfortunatelyjpropositional logic has a very limited ontology, making 
only the commitment that the world consists of facts. This made it difficult to represent even 
something as simple as the wumpus world. 

In this chapter, we examine first-order logic , 1 which makes a stronger set of ontological 
commitments. The main one is that the world consists of objects, that is, things with individual 
identities and properties that distinguish them from other objects. 

Among these objects, various relations hold. Some.-Qfdie.se relations are functions — 
relations in which there is only one "value" for a given "input." It is easy to start listing examples 
of objects, properties, relations, and functions: 

• Objects: people, houses, numbers, theories, Ronald McDonald, colors, baseball games, 
wars, centuries . . . 

• Relations: brother of, bigger than, inside, part of, has color, occurred after, owns . . . 

• Properties: red, round, bogus, prime, multistoried . . . 

. Functions: father of, best friend, third inning of, one more than . . . 

Indeed, almost any fact can be thought of as referring to objects and properties or relations. Some 
examples follow: 



• "One plus two equals three" 

Objects: one, two, three, one plus two; Relation: equals; Function: plus. (One plus two is 
a name for the object that is obtained by applying the function plus to the objects one and 
two. Three is another name for this object.) 

• "Squares neighboring the wumpus are smelly." 

Objects: wumpus, square; Property: smelly; Relation: neighboring. 

Also called first-order predicate calculus, and sometimes abbreviated as FOL or FOPC. 
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• "Evil King John ruled England in 1200.” 

Objects: John, England, 1200; Relation: ruled; Properties: evil, king. 

First-order logic has been so important to mathematics, philosophy, and artificial intelligence 
precisely because those fields — and indeed, much of everyday human existence — can be usefully 
thought of as dealing with objects and the relations between them. ,We_are not claiming that the 
world really is made up of objects and relations, just that dividing up the world that way helps us 
reason about it. First-order logic can also express facts about all of the objects in thd universeA 
This, together with the implication connective from propositional logic, enables one to represenT^ 
general laws or rules, such as the statement "Squares neighboring the wumpus are smelly." 

Although first-order logic commits to the existence of objects and relations, it does not 
make an ontological commitment to such things as categories, time, and events, which also seem 
to show up in most facts about the world. Strangely enough, this reluctance to tackle categories, 
time, and events has not hurt the popularity of first-order logic; in fact it has contributedfo its 
success. Important as these things are, there are just too many different ways to deal with them, 
and a logic that committed to a single treatment would only have limited appeal. By remaining 
neutral, first-order logic gives its users the freedom to describe these things in a way that is 
appropriate for the domain. This freedom of choice is a general characteristic of first-order logic. 

In the previous example we listed King as a property of people, but we could just as well have 
made Kircga relation between people and countries, or a function from countries to people (in a 
world in which each country has only one king). 

There are many different representation schemes in use in AI, some of which we will 
discuss in later chapters. Some are theoretically equivalent to first-order logic and some are not. 
Rutjjrstqrder logic is universal in the sense that it can express anything that can be programmed. 
We choose to study knowledge representation and reasoning using first-order logic because it 
is by far the most studied and best understood scheme yet devised. Generally speaking, other 
proposals involving additional capabilities are still hotly debated and only partially understood. 
Other proposals that are a subset of first-order logic are useful only in limited domains. Despite 
its limitations, first-order logic will be around for a long time. 



7.1 SYNTAX AND SEMANTICS 



In propositional logic every expression is a sentence, which represents a fact. First-order logic 
terms has sentences, but it also has terms, which represent objects. Constant symbols, variables-, -ajid 

function symbols are used to build terms, and quantifiers and predicate symbols are used to build 
sentences. Figure 7.1 gives a complete grammar of first-order logic, using Backus-Naur form 
(see page 854 if you are not familiar with this notation). A more detailed explanation of each 
element, describing both syntax and semantics, follows: 
constant symbols 0 Constant symbols: A, B, C, John . . . 

An interpretation must specify which object in the world is referred to by each constant 
symbol. Each constant symbol names exactly one object, but not all objects need to 
have names, and some can have several names. Thus, the symbol John, in one particular 



Section 7. 1 . Syntax and Semantics 



187 



Sentence — 



AtomicSentence 
Sentence Connective Sentence 
Quantifier Variable,. . . Sentence 
i Sentence 
(Sentence) 



AtomicSentence — Predicate(Term , . . .) Term = Term 



Term — - Function(Term , . . .) 
I Constant 
\ Variable 



Connective 

Quantifier 

Constant 

Variable 

Predicate 

Function 



— | A V | <S> 

— V | 3 

-> A\l\\ John | 

— ► Before \ HasColor \ Raining \ • • • 

— Mother \ LeftLegOf \ • • • 



Figure 7.1 The syntax of first-order logic (with equality) in BNF (Backus-Naur Form). 



interpretation, might refer to the evil King John, king of England from 1199 to 1216 
and younger brother of Richard the Lionheart. The symbol King could refer to the same 
object/person in the same interpretation. 
symbols e 0 Predicate symbols: Round, Brother, . . . 

An interpretation specifies that a predicate symbol refers to a particular relation in 
the model. For example, the Brother symbol might refer to the relation of brotherhood. 
Brother is a binary predicate symbol, and accordingly brotherhood is a relation that holds 
(or fails to hold) between pairs of objects. In any given model, the relation is defined by 
tuples the set of tuples of objects that satisfy it. A tuple is a collection of objects arranged in 

a fixed order. They are written with angle brackets surrounding the objects. In a model 
containing three objects, King John, Robin Hood, and Richard the Lionheart, the relation 
of brotherhood is defined by the set of tuples 

{ (King John, Richard the Lionheart), 

(Richard the Lionheart, King John) } 

Thus, formally speaking, Brother refers to this set of tuples under the interpretation we 
have chosen. 
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FUNCTION SYMBOLS 0 Function symbols: Cosine, FatherOf LeftLegOf .. . 

Some relations are functional — that is, any given object is related to exactly one other 
object by the relation. For example, any angle has only one number that is its cosine; any 
person has only one person that is his or her father. In such cases, it is often more convenient 
to define a function symbol (e.g., Cosine) that refers to the appropriate relation between 
angles and numbers. In the model, the mapping is just a set of n + 1 -tuples with a special 
property, namely, that the last element of each tuple is the value of the function for the 
first n elements, and each combination of the first n elements appears in exactly one tuple. 
A table of cosines is just such a set of tuples — for each possible angle of interest, it gives 
the cosine of the angle. Unlike predicate symbols, which are used to state that relations 
hold among certain objects, function symbols are used to refer to particular objects without 
using their names, as we wi 1 1 see In the next section. 

The choice of constant, predicate, and function symbols is entirely up to the user. A mathematician 
might want to use + and Cosine, a composer Crescendo and F-sharp. The names do not matter 
from a formal point of view, but it enhances readability if the intended interpretation of the 
symbols is clear. We return to this point in Section 8.1 . 



Terms 

A term is a logical expression that refers to an object. Constant symbols are therefore terms. 
Sometimes, it is more convenient to use an expression to refer to an object. For example, 
in English we might use the expression "King John's left leg" rather than giving a name to 
his leg. This is what function symbols are for: instead of using a constant symbol, we use 
Left Leg Off ohn). In the general case, a complex term is formed by a function symbol followed 
by a parenthesized list of terms as arguments to the function symbol. It is important to remember"' 
that a complex term is just a complicated kind of name. It is not a "subroutine call" that "returns 
a value." There is no LeftLegOf subroutine that takes a person as input and returns a leg. We can 
reason about left legs (e.g., stating the general rule that everyone has one and then deducing that 
John must have one) without ever providing a definition of LeftLegOf. This is something that 
cannot be done with subroutines in programming languages. \ ’ 

The formal semantics of terms is straightforward. An interpretation specifies a functional \ 
relation referred to by the function symbol, and objects referred to by the terms that are its I 
arguments. Thus, the whole term refers to the object that appears as the (n+l)-th entry in that 
tuple in the relation whose first n elements are the objects referred to by the arguments. Thus, 
the LeftLegOf function symbol might refer to the following functional relation: y* 

{ (King John, King John's leftleg), 

(Richard the Lionheart, Richard's left leg)} 

and if King John refers to King John, then LeftLegOf King Jolm ) refers, according to the relation, 
to King John's left leg. 



